# embedding_metadata

key: _node_content
string_value:
{"id_": "f1b79f14-a0d0-4898-997d-afff8d21eaa9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "71ccd574-526c-4952-a66d-468f27a84aa9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a844f363b7c467e60ce1e93bf4a95d6c56faa42cd044ecb249b3d1fc0e4a1ba", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d0bb4369-9bd9-4420-9212-614588450816", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "83ebf8945a7740152b7ed371bc95e7952f94253a79c61c3f3d95e6aea5d836c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4062e57-ca45-4cd2-a5ec-dcc8d8b67bc1", "node_type": "1", "metadata": {}, "hash": "f94a0cf9b286bf13b56756048eb29e027b83b10654d866954da7a4f982981675", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7748, "end_char_idx": 10116, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This way, when there is a bug fix, we can see all other impacted models and choose to trickle down the modification or break the copy.

<Tip>

If a file is a full copy of another file, you should register it in the constant `FULL_COPIES` of `utils/check_copies.py`.

</Tip>

This mechanism relies on comments of the form `# Copied from xxx`. The `xxx` should contain the whole path to the class of function which is being copied below. For instance, `RobertaSelfOutput` is a direct copy of the `BertSelfOutput` class, so you can see [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L289) it has a comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertSelfOutput
```

Note that instead of applying this to a whole class, you can apply it to the relevant methods that are copied from. For instance [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L598) you can see how `RobertaPreTrainedModel._init_weights` is copied from the same method in `BertPreTrainedModel` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
```

Sometimes the copy is exactly the same except for names: for instance in `RobertaAttention`, we use `RobertaSelfAttention` insted of `BertSelfAttention` but other than that, the code is exactly the same. This is why `# Copied from` supports simple string replacements with the following syntax: `Copied from xxx with foo->bar`. This means the code is copied with all instances of `foo` being replaced by `bar`. You can see how it used [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L304C1-L304C86) in `RobertaAttention` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
```

Note that there shouldn't be any spaces around the arrow (unless that space is part of the pattern to replace of course).

You can add several patterns separated by a comma. For instance here `CamemberForMaskedLM` is a direct copy of `RobertaForMaskedLM` with two replacements: `Roberta` to `Camembert` and `ROBERTA` to `CAMEMBERT`.


key: _node_content
string_value:
{"id_": "e4062e57-ca45-4cd2-a5ec-dcc8d8b67bc1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "71ccd574-526c-4952-a66d-468f27a84aa9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a844f363b7c467e60ce1e93bf4a95d6c56faa42cd044ecb249b3d1fc0e4a1ba", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f1b79f14-a0d0-4898-997d-afff8d21eaa9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "985fededb96f417f0b4e76707601d18cceb0e8ae5ce946e013605e95b1544ffe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9250, "end_char_idx": 11612, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is why `# Copied from` supports simple string replacements with the following syntax: `Copied from xxx with foo->bar`. This means the code is copied with all instances of `foo` being replaced by `bar`. You can see how it used [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L304C1-L304C86) in `RobertaAttention` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
```

Note that there shouldn't be any spaces around the arrow (unless that space is part of the pattern to replace of course).

You can add several patterns separated by a comma. For instance here `CamemberForMaskedLM` is a direct copy of `RobertaForMaskedLM` with two replacements: `Roberta` to `Camembert` and `ROBERTA` to `CAMEMBERT`. You can see [here](https://github.com/huggingface/transformers/blob/15082a9dc6950ecae63a0d3e5060b2fc7f15050a/src/transformers/models/camembert/modeling_camembert.py#L929) this is done with the comment:

```py
# Copied from transformers.models.roberta.modeling_roberta.RobertaForMaskedLM with Roberta->Camembert, ROBERTA->CAMEMBERT
```

If the order matters (because one of the replacements might conflict with a previous one), the replacements are executed from left to right.

<Tip>

If the replacements change the formatting (if you replace a short name by a very long name for instance), the copy is checked after applying the auto-formatter.

</Tip>

Another way when the patterns are just different casings of the same replacement (with an uppercased and a lowercased variants) is just to add the option `all-casing`. [Here](https://github.com/huggingface/transformers/blob/15082a9dc6950ecae63a0d3e5060b2fc7f15050a/src/transformers/models/mobilebert/modeling_mobilebert.py#L1237) is an example in `MobileBertForSequenceClassification` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertForSequenceClassification with Bert->MobileBert all-casing
```

In this case, the code is copied from `BertForSequenceClassification` by replacing:
- `Bert` by `MobileBert` (for instance when using `MobileBertModel` in the init)
- `bert` by `mobilebert` (for instance when defining `self.mobilebert`)
- `BERT` by `MOBILEBERT` (in the constant `MOBILEBERT_INPUTS_DOCSTRING`)


key: _node_content
string_value:
{"id_": "3019a3e2-f023-4440-b2c5-2a5b786a03fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tflite/", "title": "Export to TFLite", "tokens": 763, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77dd94f8-5b2f-44fd-a694-f79036e3665d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tflite/", "title": "Export to TFLite", "tokens": 763, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fdb897631f7de27ee69958805f90d2d93ea783eb379f6cf95331112798f06eb6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2358, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Export to TFLite

[TensorFlow Lite](https://www.tensorflow.org/lite/guide) is a lightweight framework for deploying machine learning models 
on resource-constrained devices, such as mobile phones, embedded systems, and Internet of Things (IoT) devices. 
TFLite is designed to optimize and run models efficiently on these devices with limited computational power, memory, and 
power consumption.
A TensorFlow Lite model is represented in a special efficient portable format identified by the `.tflite` file extension. 

🤗 Optimum offers functionality to export 🤗 Transformers models to TFLite through the `exporters.tflite` module. 
For the list of supported model architectures, please refer to [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/tflite/overview).

To export a model to TFLite, install the required dependencies:
 
```bash
pip install optimum[exporters-tf]
```

To check out all available arguments, refer to the [🤗 Optimum docs](https://huggingface.co/docs/optimum/main/en/exporters/tflite/usage_guides/export_a_model), 
or view help in command line:

```bash
optimum-cli export tflite --help
```

To export a model's checkpoint from the 🤗 Hub, for example, `google-bert/bert-base-uncased`, run the following command:

```bash
optimum-cli export tflite --model google-bert/bert-base-uncased --sequence_length 128 bert_tflite/
```

You should see the logs indicating progress and showing where the resulting `model.tflite` is saved, like this:

```bash
Validating TFLite model...
	-[✓] TFLite model output names match reference model (logits)
	- Validating TFLite Model output "logits":
		-[✓] (1, 128, 30522) matches (1, 128, 30522)
		-[x] values not close enough, max diff: 5.817413330078125e-05 (atol: 1e-05)
The TensorFlow Lite export succeeded with the warning: The maximum absolute difference between the output of the reference model and the TFLite exported model is not within the set tolerance 1e-05:
- logits: max diff = 5.817413330078125e-05.
 The exported model was saved at: bert_tflite
 ```

The example above illustrates exporting a checkpoint from 🤗 Hub. When exporting a local model, first make sure that you 
saved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the 
`local_path` to the `model` argument instead of the checkpoint name on 🤗 Hub.


key: _node_content
string_value:
{"id_": "0ec308a7-0ac8-4233-ba0c-98ea95e912be", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa96a6f4-518e-4e86-894d-10bf99217b4d", "node_type": "1", "metadata": {}, "hash": "aa61503410636d6cdc5e54d4f157ad12b345e897dc6150a6fb5978cb9be0970c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Optimize inference using torch.compile()

This guide aims to provide a benchmark on the inference speed-ups introduced with [`torch.compile()`](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) for [computer vision models in 🤗 Transformers](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers&sort=trending).

## Benefits of torch.compile
   
Depending on the model and the GPU, `torch.compile()` yields up to 30% speed-up during inference. To use `torch.compile()`, simply install any version of `torch` above 2.0. 

Compiling a model takes time, so it's useful if you are compiling the model only once instead of every time you infer.
To compile any computer vision model of your choice, call `torch.compile()` on the model as shown below:

```diff
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(MODEL_ID).to("cuda")
+ model = torch.compile(model)
```

`compile()` comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. `max-autotune` takes longer than `reduce-overhead` but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to `reduce-overhead` for inference time. In this guide, we used the default mode. You can learn more about it [here](https://pytorch.org/get-started/pytorch-2.0/#user-experience).

We benchmarked `torch.compile` with different computer vision models, tasks, types of hardware, and batch sizes on `torch` version 2.0.1.

## Benchmarking code 

Below you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time.


key: _node_content
string_value:
{"id_": "aa96a6f4-518e-4e86-894d-10bf99217b4d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ec308a7-0ac8-4233-ba0c-98ea95e912be", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "00886122e83e7aa8b918a4bec6d96f9af5ff2e1ed50192cd47e6798b0f776257", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90fe4c56-ed0e-4591-82d1-873af9b661b3", "node_type": "1", "metadata": {}, "hash": "40e9ed06e3df9126df16dc9108f1d49f0c958e748594f1fc91e8e7a63d802171", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 367, "end_char_idx": 3469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Benefits of torch.compile
   
Depending on the model and the GPU, `torch.compile()` yields up to 30% speed-up during inference. To use `torch.compile()`, simply install any version of `torch` above 2.0. 

Compiling a model takes time, so it's useful if you are compiling the model only once instead of every time you infer.
To compile any computer vision model of your choice, call `torch.compile()` on the model as shown below:

```diff
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(MODEL_ID).to("cuda")
+ model = torch.compile(model)
```

`compile()` comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. `max-autotune` takes longer than `reduce-overhead` but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to `reduce-overhead` for inference time. In this guide, we used the default mode. You can learn more about it [here](https://pytorch.org/get-started/pytorch-2.0/#user-experience).

We benchmarked `torch.compile` with different computer vision models, tasks, types of hardware, and batch sizes on `torch` version 2.0.1.

## Benchmarking code 

Below you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time.

### Image Classification with ViT

```python 
import torch
from PIL import Image
import requests
import numpy as np
from transformers import AutoImageProcessor, AutoModelForImageClassification

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224").to("cuda")
model = torch.compile(model)

processed_input = processor(image, return_tensors='pt').to(device="cuda")

with torch.no_grad():
    _ = model(**processed_input)

```

#### Object Detection with DETR

```python 
from transformers import AutoImageProcessor, AutoModelForObjectDetection

processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = AutoModelForObjectDetection.from_pretrained("facebook/detr-resnet-50").to("cuda")
model = torch.compile(model)

texts = ["a photo of a cat", "a photo of a dog"]
inputs = processor(text=texts, images=image, return_tensors="pt").to("cuda")

with torch.no_grad():
    _ = model(**inputs)
```

#### Image Segmentation with Segformer

```python 
from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation

processor = SegformerImageProcessor.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")
model = SegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512").to("cuda")
model = torch.compile(model)
seg_inputs = processor(images=image, return_tensors="pt").to("cuda")

with torch.no_grad():
    _ = model(**seg_inputs)
```

Below you can find the list of the models we benchmarked.


key: _node_content
string_value:
{"id_": "90fe4c56-ed0e-4591-82d1-873af9b661b3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa96a6f4-518e-4e86-894d-10bf99217b4d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5654cc60699ecfa2b305570fb090084068f1d981bddddd1579df50e729284437", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3864401-c038-47a8-9b0c-929548d3facb", "node_type": "1", "metadata": {}, "hash": "5325ed3cddc9941ad793c36ebc7e01c0bb28b7a5a8c55c7040a129ef35e1f35f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3471, "end_char_idx": 5599, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Image Classification** 
- [google/vit-base-patch16-224](https://huggingface.co/google/vit-base-patch16-224)
- [microsoft/beit-base-patch16-224-pt22k-ft22k](https://huggingface.co/microsoft/beit-base-patch16-224-pt22k-ft22k)
- [facebook/convnext-large-224](https://huggingface.co/facebook/convnext-large-224)
- [microsoft/resnet-50](https://huggingface.co/)

**Image Segmentation** 
- [nvidia/segformer-b0-finetuned-ade-512-512](https://huggingface.co/nvidia/segformer-b0-finetuned-ade-512-512)
- [facebook/mask2former-swin-tiny-coco-panoptic](https://huggingface.co/facebook/mask2former-swin-tiny-coco-panoptic)
- [facebook/maskformer-swin-base-ade](https://huggingface.co/facebook/maskformer-swin-base-ade)
- [google/deeplabv3_mobilenet_v2_1.0_513](https://huggingface.co/google/deeplabv3_mobilenet_v2_1.0_513)

**Object Detection** 
- [google/owlvit-base-patch32](https://huggingface.co/google/owlvit-base-patch32)
- [facebook/detr-resnet-101](https://huggingface.co/facebook/detr-resnet-101)
- [microsoft/conditional-detr-resnet-50](https://huggingface.co/microsoft/conditional-detr-resnet-50)

Below you can find visualization of inference durations with and without `torch.compile()` and percentage improvements for each model in different hardware and batch sizes. 

<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/a100_batch_comp.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_batch_comp.png" />
  </div>
   <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/t4_batch_comp.png" />
  </div>
</div>

<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_duration.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_percentage.png" />
  </div>
</div>


key: _node_content
string_value:
{"id_": "e3864401-c038-47a8-9b0c-929548d3facb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90fe4c56-ed0e-4591-82d1-873af9b661b3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "40e0e62d6a5b31b01a2939d56630306a2715d362e262ff48b2b730924104bc18", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d4dd2663-138f-41dd-9642-66f5193b9c8b", "node_type": "1", "metadata": {}, "hash": "1ee584ad0f7e7e9e75ae09c5d5c4c2876552e2eef499c4dac771ead7c1bd8ed0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4746, "end_char_idx": 7258, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/a100_batch_comp.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_batch_comp.png" />
  </div>
   <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/t4_batch_comp.png" />
  </div>
</div>

<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_duration.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_percentage.png" />
  </div>
</div>


![Duration Comparison on V100 with Batch Size of 1](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_1_duration.png)

![Percentage Improvement on T4 with Batch Size of 4](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/T4_4_percentage.png)

Below you can find inference durations in milliseconds for each model with and without `compile()`. Note that OwlViT results in OOM in larger batch sizes.

### A100 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 9.325 | 7.584 | 
| Image Segmentation/Segformer | 11.759 | 10.500 |
| Object Detection/OwlViT | 24.978 | 18.420 |
| Image Classification/BeiT | 11.282 | 8.448 | 
| Object Detection/DETR | 34.619 | 19.040 |
| Image Classification/ConvNeXT | 10.410 | 10.208 | 
| Image Classification/ResNet | 6.531 | 4.124 |
| Image Segmentation/Mask2former | 60.188 | 49.117 |
| Image Segmentation/Maskformer | 75.764 | 59.487 | 
| Image Segmentation/MobileNet | 8.583 | 3.974 |
| Object Detection/Resnet-101 | 36.276 | 18.197 |
| Object Detection/Conditional-DETR | 31.219 | 17.993 |


### A100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 14.832 | 14.499 | 
| Image Segmentation/Segformer | 18.838 | 16.476 |
| Image Classification/BeiT | 13.205 | 13.048 | 
| Object Detection/DETR | 48.657 | 32.418|
| Image Classification/ConvNeXT | 22.940 | 21.631 | 
| Image Classification/ResNet | 6.657 | 4.


key: _node_content
string_value:
{"id_": "d4dd2663-138f-41dd-9642-66f5193b9c8b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3864401-c038-47a8-9b0c-929548d3facb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4d866499662ed3e24e734859baf1180f7bdc272e30d0ba5175764f85e5fcc439", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd62f1cc-3981-4779-9905-cd7702fc29dc", "node_type": "1", "metadata": {}, "hash": "d6b7857f59fc78b1b22c907248693b118eb20152ccf5e14f5bcd2463ebb449eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6844, "end_char_idx": 8889, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### A100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 14.832 | 14.499 | 
| Image Segmentation/Segformer | 18.838 | 16.476 |
| Image Classification/BeiT | 13.205 | 13.048 | 
| Object Detection/DETR | 48.657 | 32.418|
| Image Classification/ConvNeXT | 22.940 | 21.631 | 
| Image Classification/ResNet | 6.657 | 4.268 |
| Image Segmentation/Mask2former | 74.277 | 61.781 |
| Image Segmentation/Maskformer | 180.700 | 159.116 | 
| Image Segmentation/MobileNet | 14.174 | 8.515 |
| Object Detection/Resnet-101 | 68.101 | 44.998 |
| Object Detection/Conditional-DETR | 56.470 | 35.552 |

### A100 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 40.944 | 40.010 | 
| Image Segmentation/Segformer | 37.005 | 31.144 |
| Image Classification/BeiT | 41.854 | 41.048 | 
| Object Detection/DETR | 164.382 | 161.902 |
| Image Classification/ConvNeXT | 82.258 | 75.561 | 
| Image Classification/ResNet | 7.018 | 5.024 |
| Image Segmentation/Mask2former | 178.945 | 154.814 |
| Image Segmentation/Maskformer | 638.570 | 579.826 | 
| Image Segmentation/MobileNet | 51.693 | 30.310 |
| Object Detection/Resnet-101 | 232.887 | 155.021 |
| Object Detection/Conditional-DETR | 180.491 | 124.032 |

### V100 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 10.495 | 6.00 | 
| Image Segmentation/Segformer | 13.321 | 5.862 | 
| Object Detection/OwlViT | 25.769 | 22.395 | 
| Image Classification/BeiT | 11.347 | 7.234 | 
| Object Detection/DETR | 33.951 | 19.388 |
| Image Classification/ConvNeXT | 11.623 | 10.412 | 
| Image Classification/ResNet | 6.484 | 3.820 |
| Image Segmentation/Mask2former | 64.640 | 49.873 |
| Image Segmentation/Maskformer | 95.532 | 72.207 | 
| Image Segmentation/MobileNet | 9.217 | 4.753 |
| Object Detection/Resnet-101 | 52.818 | 28.


key: _node_content
string_value:
{"id_": "fd62f1cc-3981-4779-9905-cd7702fc29dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d4dd2663-138f-41dd-9642-66f5193b9c8b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a23dccad667f4900de55626c110a5eb0b33acfccbfc24f557c4f5b59f21d14c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9fce1f4-2dac-4e4b-b920-d21ce5ba228a", "node_type": "1", "metadata": {}, "hash": "351268f5efc603dca17cf46b34f34824d971c976bcc080d2a098e9d731fe7789", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7836, "end_char_idx": 9911, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
382 | 161.902 |
| Image Classification/ConvNeXT | 82.258 | 75.561 | 
| Image Classification/ResNet | 7.018 | 5.024 |
| Image Segmentation/Mask2former | 178.945 | 154.814 |
| Image Segmentation/Maskformer | 638.570 | 579.826 | 
| Image Segmentation/MobileNet | 51.693 | 30.310 |
| Object Detection/Resnet-101 | 232.887 | 155.021 |
| Object Detection/Conditional-DETR | 180.491 | 124.032 |

### V100 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 10.495 | 6.00 | 
| Image Segmentation/Segformer | 13.321 | 5.862 | 
| Object Detection/OwlViT | 25.769 | 22.395 | 
| Image Classification/BeiT | 11.347 | 7.234 | 
| Object Detection/DETR | 33.951 | 19.388 |
| Image Classification/ConvNeXT | 11.623 | 10.412 | 
| Image Classification/ResNet | 6.484 | 3.820 |
| Image Segmentation/Mask2former | 64.640 | 49.873 |
| Image Segmentation/Maskformer | 95.532 | 72.207 | 
| Image Segmentation/MobileNet | 9.217 | 4.753 |
| Object Detection/Resnet-101 | 52.818 | 28.367 |
| Object Detection/Conditional-DETR | 39.512 | 20.816 |

### V100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 15.181 | 14.501 | 
| Image Segmentation/Segformer | 16.787 | 16.188 |
| Image Classification/BeiT | 15.171 | 14.753 | 
| Object Detection/DETR | 88.529 | 64.195 |
| Image Classification/ConvNeXT | 29.574 | 27.085 | 
| Image Classification/ResNet | 6.109 | 4.731 |
| Image Segmentation/Mask2former | 90.402 | 76.926 |
| Image Segmentation/Maskformer | 234.261 | 205.456 | 
| Image Segmentation/MobileNet | 24.623 | 14.816 |
| Object Detection/Resnet-101 | 134.672 | 101.304 |
| Object Detection/Conditional-DETR | 97.464 | 69.739 |

### V100 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 52.209 | 51.633 | 
| Image Segmentation/Segformer | 61.013 | 55.499 |
| Image Classification/BeiT | 53.938 | 53.


key: _node_content
string_value:
{"id_": "a9fce1f4-2dac-4e4b-b920-d21ce5ba228a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd62f1cc-3981-4779-9905-cd7702fc29dc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4750ee13b8f6d67b7f15cd1dbc92382e1599a94ed5a0b375e54a4a5710ec75d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "85dad5b9-516b-411d-9866-8b2f2bed89b0", "node_type": "1", "metadata": {}, "hash": "4f31c039172791eb9fb95cd3aeb5850d0eaeafde8a5db06dcac79c14a7b7b692", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8880, "end_char_idx": 10950, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
818 | 28.367 |
| Object Detection/Conditional-DETR | 39.512 | 20.816 |

### V100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 15.181 | 14.501 | 
| Image Segmentation/Segformer | 16.787 | 16.188 |
| Image Classification/BeiT | 15.171 | 14.753 | 
| Object Detection/DETR | 88.529 | 64.195 |
| Image Classification/ConvNeXT | 29.574 | 27.085 | 
| Image Classification/ResNet | 6.109 | 4.731 |
| Image Segmentation/Mask2former | 90.402 | 76.926 |
| Image Segmentation/Maskformer | 234.261 | 205.456 | 
| Image Segmentation/MobileNet | 24.623 | 14.816 |
| Object Detection/Resnet-101 | 134.672 | 101.304 |
| Object Detection/Conditional-DETR | 97.464 | 69.739 |

### V100 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 52.209 | 51.633 | 
| Image Segmentation/Segformer | 61.013 | 55.499 |
| Image Classification/BeiT | 53.938 | 53.581  |
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 109.682 | 100.771 | 
| Image Classification/ResNet | 14.857 | 12.089 |
| Image Segmentation/Mask2former | 249.605 | 222.801 |
| Image Segmentation/Maskformer | 831.142 | 743.645 | 
| Image Segmentation/MobileNet | 93.129 | 55.365 |
| Object Detection/Resnet-101 | 482.425 | 361.843 |
| Object Detection/Conditional-DETR | 344.661 | 255.298 |

### T4 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 16.520 | 15.786 | 
| Image Segmentation/Segformer | 16.116 | 14.205 |
| Object Detection/OwlViT | 53.634 | 51.105 |
| Image Classification/BeiT | 16.464 | 15.710 | 
| Object Detection/DETR | 73.100 | 53.99 |
| Image Classification/ConvNeXT | 32.932 | 30.845 | 
| Image Classification/ResNet | 6.031 | 4.321 |
| Image Segmentation/Mask2former | 79.192 | 66.815 |
| Image Segmentation/Maskformer | 200.026 | 188.268 | 
| Image Segmentation/MobileNet | 18.908 | 11.


key: _node_content
string_value:
{"id_": "85dad5b9-516b-411d-9866-8b2f2bed89b0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9fce1f4-2dac-4e4b-b920-d21ce5ba228a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "97231fc6287c6860cf1515b401efb963cfb42538d238d72fe299a187a49b389d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "91b784cd-947c-4b98-8693-c781acf77694", "node_type": "1", "metadata": {}, "hash": "a5e248a607ec8791392e67a56ac6aa4d5ecd033ff6543db0de1bdaa177e84087", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9863, "end_char_idx": 11975, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
499 |
| Image Classification/BeiT | 53.938 | 53.581  |
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 109.682 | 100.771 | 
| Image Classification/ResNet | 14.857 | 12.089 |
| Image Segmentation/Mask2former | 249.605 | 222.801 |
| Image Segmentation/Maskformer | 831.142 | 743.645 | 
| Image Segmentation/MobileNet | 93.129 | 55.365 |
| Object Detection/Resnet-101 | 482.425 | 361.843 |
| Object Detection/Conditional-DETR | 344.661 | 255.298 |

### T4 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 16.520 | 15.786 | 
| Image Segmentation/Segformer | 16.116 | 14.205 |
| Object Detection/OwlViT | 53.634 | 51.105 |
| Image Classification/BeiT | 16.464 | 15.710 | 
| Object Detection/DETR | 73.100 | 53.99 |
| Image Classification/ConvNeXT | 32.932 | 30.845 | 
| Image Classification/ResNet | 6.031 | 4.321 |
| Image Segmentation/Mask2former | 79.192 | 66.815 |
| Image Segmentation/Maskformer | 200.026 | 188.268 | 
| Image Segmentation/MobileNet | 18.908 | 11.997 |
| Object Detection/Resnet-101 | 106.622 | 82.566 |
| Object Detection/Conditional-DETR | 77.594 | 56.984 |

### T4 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 43.653 | 43.626 | 
| Image Segmentation/Segformer | 45.327 | 42.445 |
| Image Classification/BeiT | 52.007 | 51.354 | 
| Object Detection/DETR | 277.850 | 268.003 |
| Image Classification/ConvNeXT | 119.259 | 105.580 | 
| Image Classification/ResNet | 13.039 | 11.388 |
| Image Segmentation/Mask2former | 201.540 | 184.670 |
| Image Segmentation/Maskformer | 764.052 | 711.280 | 
| Image Segmentation/MobileNet | 74.289 | 48.677 |
| Object Detection/Resnet-101 | 421.859 | 357.614 |
| Object Detection/Conditional-DETR | 289.002 | 226.945 |

### T4 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 163.914 | 160.907 | 
| Image Segmentation/Segformer | 192.


key: _node_content
string_value:
{"id_": "91b784cd-947c-4b98-8693-c781acf77694", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "85dad5b9-516b-411d-9866-8b2f2bed89b0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "69ac048efb67d8bb1f7a2ca70c87b7cf528e7bda191723c4dd7da4f705b26939", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68f6a092-1ca0-4a4b-a020-d93e7866cbba", "node_type": "1", "metadata": {}, "hash": "a9f9b97c53781c583bd40baa27483b6c8dd5a0cbb4cbb2e48db9ee7bacd832a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10941, "end_char_idx": 12680, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
908 | 11.997 |
| Object Detection/Resnet-101 | 106.622 | 82.566 |
| Object Detection/Conditional-DETR | 77.594 | 56.984 |

### T4 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 43.653 | 43.626 | 
| Image Segmentation/Segformer | 45.327 | 42.445 |
| Image Classification/BeiT | 52.007 | 51.354 | 
| Object Detection/DETR | 277.850 | 268.003 |
| Image Classification/ConvNeXT | 119.259 | 105.580 | 
| Image Classification/ResNet | 13.039 | 11.388 |
| Image Segmentation/Mask2former | 201.540 | 184.670 |
| Image Segmentation/Maskformer | 764.052 | 711.280 | 
| Image Segmentation/MobileNet | 74.289 | 48.677 |
| Object Detection/Resnet-101 | 421.859 | 357.614 |
| Object Detection/Conditional-DETR | 289.002 | 226.945 |

### T4 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 163.914 | 160.907 | 
| Image Segmentation/Segformer | 192.412 | 163.620 |
| Image Classification/BeiT | 188.978 | 187.976 | 
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 422.886 | 388.078 | 
| Image Classification/ResNet | 44.114 | 37.604 |
| Image Segmentation/Mask2former | 756.337 | 695.291 |
| Image Segmentation/Maskformer | 2842.940 | 2656.88 | 
| Image Segmentation/MobileNet | 299.003 | 201.942 |
| Object Detection/Resnet-101 |  1619.505 | 1262.758 | 
| Object Detection/Conditional-DETR | 1137.513 | 897.390|

## PyTorch Nightly
We also benchmarked on PyTorch nightly (2.1.0dev, find the wheel [here](https://download.pytorch.org/whl/nightly/cu118)) and observed improvement in latency both for uncompiled and compiled models.


key: _node_content
string_value:
{"id_": "68f6a092-1ca0-4a4b-a020-d93e7866cbba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "91b784cd-947c-4b98-8693-c781acf77694", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9b5bb9db4cebccf875d88d1bae48eeaefdbbd7fc8b65b3867581a5eabb6439c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dfbea294-cfa3-4f93-9d0f-6d993c289582", "node_type": "1", "metadata": {}, "hash": "9192f96ee7c0c80ad4248351255391c240126ba3a4eda7f3d7e4a8853ce4b890", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11538, "end_char_idx": 14041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
670 |
| Image Segmentation/Maskformer | 764.052 | 711.280 | 
| Image Segmentation/MobileNet | 74.289 | 48.677 |
| Object Detection/Resnet-101 | 421.859 | 357.614 |
| Object Detection/Conditional-DETR | 289.002 | 226.945 |

### T4 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 163.914 | 160.907 | 
| Image Segmentation/Segformer | 192.412 | 163.620 |
| Image Classification/BeiT | 188.978 | 187.976 | 
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 422.886 | 388.078 | 
| Image Classification/ResNet | 44.114 | 37.604 |
| Image Segmentation/Mask2former | 756.337 | 695.291 |
| Image Segmentation/Maskformer | 2842.940 | 2656.88 | 
| Image Segmentation/MobileNet | 299.003 | 201.942 |
| Object Detection/Resnet-101 |  1619.505 | 1262.758 | 
| Object Detection/Conditional-DETR | 1137.513 | 897.390|

## PyTorch Nightly
We also benchmarked on PyTorch nightly (2.1.0dev, find the wheel [here](https://download.pytorch.org/whl/nightly/cu118)) and observed improvement in latency both for uncompiled and compiled models. 

### A100

| **Task/Model** | **Batch Size** | **torch 2.0 - no compile** | **torch 2.0 -<br> compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/BeiT | Unbatched | 12.462 | 6.954 | 
| Image Classification/BeiT | 4 | 14.109 | 12.851 | 
| Image Classification/BeiT | 16 | 42.179 | 42.147 | 
| Object Detection/DETR | Unbatched | 30.484 | 15.221 |
| Object Detection/DETR | 4 | 46.816 | 30.942 |
| Object Detection/DETR | 16 | 163.749 | 163.706  |

### T4

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/BeiT | Unbatched | 14.408 | 14.052 | 
| Image Classification/BeiT | 4 | 47.381 | 46.604 | 
| Image Classification/BeiT | 16 | 42.179 | 42.147  | 
| Object Detection/DETR | Unbatched | 68.382 | 53.481 |
| Object Detection/DETR | 4 | 269.615 | 204.785 |
| Object Detection/DETR | 16 | OOM | OOM   |

### V100

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/BeiT | Unbatched | 13.477 | 7.926 | 
| Image Classification/BeiT | 4 | 15.103 | 14.378 | 
| Image Classification/BeiT | 16 | 52.517 | 51.691  | 
| Object Detection/DETR | Unbatched | 28.706 | 19.077 |
| Object Detection/DETR | 4 | 88.402 | 62.949|
| Object Detection/DETR | 16 | OOM | OOM  |


key: _node_content
string_value:
{"id_": "dfbea294-cfa3-4f93-9d0f-6d993c289582", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0694b540-71d3-4127-b1be-89e1cf12c6ff", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "abf327d9d2c9432cdfe04a57a613ff60b012fad7d31035ef7d65d39f8555a9d9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68f6a092-1ca0-4a4b-a020-d93e7866cbba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile/", "title": "Optimize inference using torch.compile()", "tokens": 5179, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2c2eb5a90c6034aa069cd4b5d1012e457c517d68aba0c3a0c123c2e7db622811", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14044, "end_char_idx": 15386, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Reduce Overhead
We benchmarked `reduce-overhead` compilation mode for A100 and T4 in Nightly.

### A100

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/ConvNeXT | Unbatched | 11.758 | 7.335 | 
| Image Classification/ConvNeXT | 4 | 23.171 | 21.490 | 
| Image Classification/ResNet | Unbatched | 7.435 | 3.801 | 
| Image Classification/ResNet | 4 | 7.261 | 2.187 | 
| Object Detection/Conditional-DETR | Unbatched | 32.823 | 11.627  | 
| Object Detection/Conditional-DETR | 4 | 50.622 | 33.831  | 
| Image Segmentation/MobileNet | Unbatched | 9.869 | 4.244 |
| Image Segmentation/MobileNet | 4 | 14.385 | 7.946 |


### T4

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** | 
|:---:|:---:|:---:|:---:|
| Image Classification/ConvNeXT | Unbatched | 32.137 | 31.84 | 
| Image Classification/ConvNeXT | 4 | 120.944 | 110.209 | 
| Image Classification/ResNet | Unbatched | 9.761 | 7.698 | 
| Image Classification/ResNet | 4 | 15.215 | 13.871 | 
| Object Detection/Conditional-DETR | Unbatched | 72.150 | 57.660  | 
| Object Detection/Conditional-DETR | 4 | 301.494 | 247.543  | 
| Image Segmentation/MobileNet | Unbatched | 22.266 | 19.339  |
| Image Segmentation/MobileNet | 4 | 78.311 | 50.983 |


key: _node_content
string_value:
{"id_": "ebbc1564-b30a-4ff8-b664-71cdcf249843", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_special/", "title": "PyTorch training on Apple silicon", "tokens": 771, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "66c01833-3f30-4cf2-b91e-9157bf2fbadf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_special/", "title": "PyTorch training on Apple silicon", "tokens": 771, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "97e01e2d8466584de90545a9ff4cc25679d1e02be781cec754fc40bc47acd214", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PyTorch training on Apple silicon

Previously, training models on a Mac was limited to the CPU only. With the release of PyTorch v1.12, you can take advantage of training models with Apple's silicon GPUs for significantly faster performance and training. This is powered in PyTorch by integrating Apple's Metal Performance Shaders (MPS) as a backend. The [MPS backend](https://pytorch.org/docs/stable/notes/mps.html) implements PyTorch operations as custom Metal shaders and places these modules on a `mps` device.

<Tip warning={true}>

Some PyTorch operations are not implemented in MPS yet and will throw an error. To avoid this, you should set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU kernels instead (you'll still see a `UserWarning`).

<br>

If you run into any other errors, please open an issue in the [PyTorch](https://github.com/pytorch/pytorch/issues) repository because the [`Trainer`] only integrates the MPS backend.

</Tip>

With the `mps` device set, you can:

* train larger networks or batch sizes locally
* reduce data retrieval latency because the GPU's unified memory architecture allows direct access to the full memory store
* reduce costs because you don't need to train on cloud-based GPUs or add additional local GPUs

Get started by making sure you have PyTorch installed. MPS acceleration is supported on macOS 12.3+.

```bash
pip install torch torchvision torchaudio
```

[`TrainingArguments`] uses the `mps` device by default if it's available which means you don't need to explicitly set the device. For example, you can run the [run_glue.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py) script with the MPS backend automatically enabled without making any changes.

```diff
export TASK_NAME=mrpc

python examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path google-bert/bert-base-cased \
  --task_name $TASK_NAME \
- --use_mps_device \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --overwrite_output_dir
```

Backends for [distributed setups](https://pytorch.org/docs/stable/distributed.html#backends) like `gloo` and `nccl` are not supported by the `mps` device which means you can only train on a single GPU with the MPS backend.

You can learn more about the MPS backend in the [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/) blog post.


key: _node_content
string_value:
{"id_": "0c802de9-94d5-4ade-8720-c9b922afafc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f874371-31d1-4fb6-9879-48341dd1b849", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c9d5ebe9a87ec31ed848f01b119034ad556a396beeff3ff6301cb1ca2e7487ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb1c0287-329d-4dfc-a582-0e518fd5d2a2", "node_type": "1", "metadata": {}, "hash": "e27405c2aa0699f084d144afa325ddd58ce5fd51e8c95ec25e52f539539dd320", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Load adapters with 🤗 PEFT

[[open-in-colab]]

[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it. The adapters are trained to learn task-specific information. This approach has been shown to be very memory-efficient with lower compute usage while producing results comparable to a fully fine-tuned model. 

Adapters trained with PEFT are also usually an order of magnitude smaller than the full model, making it convenient to share, store, and load them.

<div class="flex flex-col justify-center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png"/>
  <figcaption class="text-center">The adapter weights for a OPTForCausalLM model stored on the Hub are only ~6MB compared to the full size of the model weights, which can be ~700MB.</figcaption>
</div>

If you're interested in learning more about the 🤗 PEFT library, check out the [documentation](https://huggingface.co/docs/peft/index).

## Setup

Get started by installing 🤗 PEFT:

```bash
pip install peft
```

If you want to try out the brand new features, you might be interested in installing the library from source:

```bash
pip install git+https://github.com/huggingface/peft.git
```

## Supported PEFT models

🤗 Transformers natively supports some PEFT methods, meaning you can load adapter weights stored locally or on the Hub and easily run or train them with a few lines of code. The following methods are supported:

- [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora)
- [IA3](https://huggingface.co/docs/peft/conceptual_guides/ia3)
- [AdaLoRA](https://arxiv.org/abs/2303.10512)

If you want to use other PEFT methods, such as prompt learning or prompt tuning, or about the 🤗 PEFT library in general, please refer to the [documentation](https://huggingface.co/docs/peft/index).


## Load a PEFT adapter

To load and use a PEFT adapter model from 🤗 Transformers, make sure the Hub repository or local directory contains an `adapter_config.json` file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the `AutoModelFor` class. For example, to load a PEFT adapter model for causal language modeling:

1. specify the PEFT model id
2. pass it to the [`AutoModelForCausalLM`] class

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id)
```

<Tip>

You can load a PEFT adapter with either an `AutoModelFor` class or the base model class like `OPTForCausalLM` or `LlamaForCausalLM`.


key: _node_content
string_value:
{"id_": "fb1c0287-329d-4dfc-a582-0e518fd5d2a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f874371-31d1-4fb6-9879-48341dd1b849", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c9d5ebe9a87ec31ed848f01b119034ad556a396beeff3ff6301cb1ca2e7487ab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c802de9-94d5-4ade-8720-c9b922afafc6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bb36f4ccd43d472bd29b6de835df5d94b5008147fea1a5d4fc1d6123c7202c8c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5da2418e-629f-4594-a72b-c9c540ebd918", "node_type": "1", "metadata": {}, "hash": "4439f1bbbb111b77b87e2ed770a99c44b135919652977be5dc179b7c81383eb8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2009, "end_char_idx": 4089, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Load a PEFT adapter

To load and use a PEFT adapter model from 🤗 Transformers, make sure the Hub repository or local directory contains an `adapter_config.json` file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the `AutoModelFor` class. For example, to load a PEFT adapter model for causal language modeling:

1. specify the PEFT model id
2. pass it to the [`AutoModelForCausalLM`] class

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id)
```

<Tip>

You can load a PEFT adapter with either an `AutoModelFor` class or the base model class like `OPTForCausalLM` or `LlamaForCausalLM`.

</Tip>

You can also load a PEFT adapter by calling the `load_adapter` method:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "facebook/opt-350m"
peft_model_id = "ybelkada/opt-350m-lora"

model = AutoModelForCausalLM.from_pretrained(model_id)
model.load_adapter(peft_model_id)
```

Check out the [API documentation](#transformers.integrations.PeftAdapterMixin) section below for more details.

## Load in 8bit or 4bit

The `bitsandbytes` integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the `bitsandbytes` integration [guide](./quantization#bitsandbytes-integration) to learn more). Add the `load_in_8bit` or `load_in_4bit` parameters to [`~PreTrainedModel.from_pretrained`] and set `device_map="auto"` to effectively distribute the model to your hardware:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

## Add a new adapter

You can use [`~peft.PeftModel.add_adapter`] to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one.


key: _node_content
string_value:
{"id_": "5da2418e-629f-4594-a72b-c9c540ebd918", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f874371-31d1-4fb6-9879-48341dd1b849", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c9d5ebe9a87ec31ed848f01b119034ad556a396beeff3ff6301cb1ca2e7487ab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fb1c0287-329d-4dfc-a582-0e518fd5d2a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d9191eaddcdf4d4333f917bd2858fda3ff89fb46ec1d93e0751582e8567b8c9a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8703479d-abff-498f-9101-1ee518705ff3", "node_type": "1", "metadata": {}, "hash": "e02d31294110cd2094935b3338e93851a0aa7fbf820db5e4b4ba6744047aae34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2780, "end_char_idx": 5167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

You can also load a PEFT adapter by calling the `load_adapter` method:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "facebook/opt-350m"
peft_model_id = "ybelkada/opt-350m-lora"

model = AutoModelForCausalLM.from_pretrained(model_id)
model.load_adapter(peft_model_id)
```

Check out the [API documentation](#transformers.integrations.PeftAdapterMixin) section below for more details.

## Load in 8bit or 4bit

The `bitsandbytes` integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the `bitsandbytes` integration [guide](./quantization#bitsandbytes-integration) to learn more). Add the `load_in_8bit` or `load_in_4bit` parameters to [`~PreTrainedModel.from_pretrained`] and set `device_map="auto"` to effectively distribute the model to your hardware:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

## Add a new adapter

You can use [`~peft.PeftModel.add_adapter`] to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one. For example, if you have an existing LoRA adapter attached to a model:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    init_lora_weights=False
)

model.add_adapter(lora_config, adapter_name="adapter_1")
```

To add a new adapter:

```py
# attach new adapter with same config
model.add_adapter(lora_config, adapter_name="adapter_2")
```

Now you can use [`~peft.PeftModel.set_adapter`] to set which adapter to use:

```py
# use adapter_1
model.set_adapter("adapter_1")
output = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))

# use adapter_2
model.set_adapter("adapter_2")
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
```

## Enable and disable adapters

Once you've added an adapter to a model, you can enable or disable the adapter module.


key: _node_content
string_value:
{"id_": "8703479d-abff-498f-9101-1ee518705ff3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f874371-31d1-4fb6-9879-48341dd1b849", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c9d5ebe9a87ec31ed848f01b119034ad556a396beeff3ff6301cb1ca2e7487ab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5da2418e-629f-4594-a72b-c9c540ebd918", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2fa64e3c5cf2547ad4ad76e17b2785dce56fd54627c1802144bff3cabc72aee4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf3a696c-638c-4da6-84a5-4f1308964f9d", "node_type": "1", "metadata": {}, "hash": "8bc89707a9d994367f824d0c23884bd9c696dc335ef2a4491d941c57e222aafa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4090, "end_char_idx": 7112, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have an existing LoRA adapter attached to a model:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    init_lora_weights=False
)

model.add_adapter(lora_config, adapter_name="adapter_1")
```

To add a new adapter:

```py
# attach new adapter with same config
model.add_adapter(lora_config, adapter_name="adapter_2")
```

Now you can use [`~peft.PeftModel.set_adapter`] to set which adapter to use:

```py
# use adapter_1
model.set_adapter("adapter_1")
output = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))

# use adapter_2
model.set_adapter("adapter_2")
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
```

## Enable and disable adapters

Once you've added an adapter to a model, you can enable or disable the adapter module. To enable the adapter module:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig

model_id = "facebook/opt-350m"
adapter_model_id = "ybelkada/opt-350m-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Hello"
inputs = tokenizer(text, return_tensors="pt")

model = AutoModelForCausalLM.from_pretrained(model_id)
peft_config = PeftConfig.from_pretrained(adapter_model_id)

# to initiate with random weights
peft_config.init_lora_weights = False

model.add_adapter(peft_config)
model.enable_adapters()
output = model.generate(**inputs)
```

To disable the adapter module:

```py
model.disable_adapters()
output = model.generate(**inputs)
```

## Train a PEFT adapter

PEFT adapters are supported by the [`Trainer`] class so that you can train an adapter for your specific use case. It only requires adding a few more lines of code. For example, to train a LoRA adapter:

<Tip>

If you aren't familiar with fine-tuning a model with [`Trainer`], take a look at the [Fine-tune a pretrained model](training) tutorial.

</Tip>

1. Define your adapter configuration with the task type and hyperparameters (see [`~peft.LoraConfig`] for more details about what the hyperparameters do).

```py
from peft import LoraConfig

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)
```

2. Add adapter to the model.

```py
model.add_adapter(peft_config)
```

3. Now you can pass the model to [`Trainer`]!

```py
trainer = Trainer(model=model, ...)
trainer.train()
```

To save your trained adapter and load it back:

```py
model.save_pretrained(save_dir)
model = AutoModelForCausalLM.from_pretrained(save_dir)
```

## Add additional trainable layers to a PEFT adapter

You can also fine-tune additional trainable adapters on top of a model that has adapters attached by passing `modules_to_save` in your PEFT config.


key: _node_content
string_value:
{"id_": "cf3a696c-638c-4da6-84a5-4f1308964f9d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f874371-31d1-4fb6-9879-48341dd1b849", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c9d5ebe9a87ec31ed848f01b119034ad556a396beeff3ff6301cb1ca2e7487ab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8703479d-abff-498f-9101-1ee518705ff3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft/", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9231235c5cd8fe1d4e7a6fa91a302fb583de7cd1c0321baf186d578d297c5b6d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6023, "end_char_idx": 7882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It only requires adding a few more lines of code. For example, to train a LoRA adapter:

<Tip>

If you aren't familiar with fine-tuning a model with [`Trainer`], take a look at the [Fine-tune a pretrained model](training) tutorial.

</Tip>

1. Define your adapter configuration with the task type and hyperparameters (see [`~peft.LoraConfig`] for more details about what the hyperparameters do).

```py
from peft import LoraConfig

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)
```

2. Add adapter to the model.

```py
model.add_adapter(peft_config)
```

3. Now you can pass the model to [`Trainer`]!

```py
trainer = Trainer(model=model, ...)
trainer.train()
```

To save your trained adapter and load it back:

```py
model.save_pretrained(save_dir)
model = AutoModelForCausalLM.from_pretrained(save_dir)
```

## Add additional trainable layers to a PEFT adapter

You can also fine-tune additional trainable adapters on top of a model that has adapters attached by passing `modules_to_save` in your PEFT config. For example, if you want to also fine-tune the lm_head on top of a model with a LoRA adapter:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    modules_to_save=["lm_head"],
)

model.add_adapter(lora_config)
```

## API docs

[[autodoc]] integrations.PeftAdapterMixin
    - load_adapter
    - add_adapter
    - set_adapter
    - disable_adapters
    - enable_adapters
    - active_adapters
    - get_adapter_state_dict




<!--
TODO: (@younesbelkada @stevhliu)
-   Link to PEFT docs for further details
-   Trainer  
-   8-bit / 4-bit examples ?
-->


key: _node_content
string_value:
{"id_": "2705d5d3-baf6-4b3a-a36a-a5dd61166ae0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43c9d2a9-9656-4973-9c9b-eef595f31648", "node_type": "1", "metadata": {}, "hash": "36c01562d2d54924e66a1d73f7069c5b734782af77a28d1c786cef6dc8296979", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Translation

[[open-in-colab]]

<Youtube id="1JvfrvZgi6c"/>

Translation converts a sequence of text from one language to another. It is one of several tasks you can formulate as a sequence-to-sequence problem, a powerful framework for returning some output from an input, like translation or summarization. Translation systems are commonly used for translation between different language texts, but it can also be used for speech or some combination in between like text-to-speech or speech-to-text.

This guide will show you how to:

1. Finetune [T5](https://huggingface.co/google-t5/t5-small) on the English-French subset of the [OPUS Books](https://huggingface.co/datasets/opus_books) dataset to translate English text to French.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/translation).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate sacrebleu
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load OPUS Books dataset

Start by loading the English-French subset of the [OPUS Books](https://huggingface.co/datasets/opus_books) dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> books = load_dataset("opus_books", "en-fr")
```

Split the dataset into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> books = books["train"].train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> books["train"][0]
{'id': '90560',
 'translation': {'en': 'But this lofty plateau measured only a few fathoms, and soon we reentered Our Element.',
  'fr': 'Mais ce plateau élevé ne mesurait que quelques toises, et bientôt nous fûmes rentrés dans notre élément.'}}
```

`translation`: an English and French translation of the text.

## Preprocess

<Youtube id="XAR8jnZZuUs"/>

The next step is to load a T5 tokenizer to process the English-French language pairs:

```py
>>> from transformers import AutoTokenizer

>>> checkpoint = "google-t5/t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
```

The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a translation task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Tokenize the input (English) and target (French) separately because you can't tokenize French text with a tokenizer pretrained on an English vocabulary.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
>>> source_lang = "en"
>>> target_lang = "fr"
>>> prefix = "translate English to French: "


key: _node_content
string_value:
{"id_": "43c9d2a9-9656-4973-9c9b-eef595f31648", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2705d5d3-baf6-4b3a-a36a-a5dd61166ae0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9586b24f802c434b38f938ddd0969ac9bf0e73064577c1cf4c3de62c45720e98", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b4a08417-556b-42ee-b0f1-e51878178faf", "node_type": "1", "metadata": {}, "hash": "f424f901e7a4e1f207adbbec5fcc63901bddeae43953d7efa8b2d8c73cd3a6aa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2988, "end_char_idx": 6244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def preprocess_function(examples):
...     inputs = [prefix + example[source_lang] for example in examples["translation"]]
...     targets = [example[target_lang] for example in examples["translation"]]
...     model_inputs = tokenizer(inputs, text_target=targets, max_length=128, truncation=True)
...     return model_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_books = books.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorForSeq2Seq`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)
```
</pt>
<tf>

```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [SacreBLEU](https://huggingface.co/spaces/evaluate-metric/sacrebleu) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> metric = evaluate.load("sacrebleu")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the SacreBLEU score:

```py
>>> import numpy as np


>>> def postprocess_text(preds, labels):
...     preds = [pred.strip() for pred in preds]
...     labels = [[label.strip()] for label in labels]

...     return preds, labels


>>> def compute_metrics(eval_preds):
...     preds, labels = eval_preds
...     if isinstance(preds, tuple):
...         preds = preds[0]
...     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

...     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

...     result = metric.compute(predictions=decoded_preds, references=decoded_labels)
...     result = {"bleu": result["score"]}

...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
...     result["gen_len"] = np.mean(prediction_lens)
...     result = {k: round(v, 4) for k, v in result.items()}
...     return result
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!


key: _node_content
string_value:
{"id_": "b4a08417-556b-42ee-b0f1-e51878178faf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43c9d2a9-9656-4973-9c9b-eef595f31648", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0018e91a9d370c0c24de0a49e0f45f7118fbc2906d84548ad49082c273483944", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c51b997-8632-494a-a2be-c1b9fbf78836", "node_type": "1", "metadata": {}, "hash": "b908af4f159c3e551b2b0af5e0c38554a20b4d7b9cf208c0c62e066f5fd9ce94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4938, "end_char_idx": 7141, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def postprocess_text(preds, labels):
...     preds = [pred.strip() for pred in preds]
...     labels = [[label.strip()] for label in labels]

...     return preds, labels


>>> def compute_metrics(eval_preds):
...     preds, labels = eval_preds
...     if isinstance(preds, tuple):
...         preds = preds[0]
...     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

...     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

...     result = metric.compute(predictions=decoded_preds, references=decoded_labels)
...     result = {"bleu": result["score"]}

...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
...     result["gen_len"] = np.mean(prediction_lens)
...     result = {k: round(v, 4) for k, v in result.items()}
...     return result
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:

```py
>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "8c51b997-8632-494a-a2be-c1b9fbf78836", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b4a08417-556b-42ee-b0f1-e51878178faf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "445daffbbd5ec94a60291a9bb7a428515397b823e6be274f7cb0c78e437cf34b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cfabeaa1-10f0-474c-ac3f-51684ebaecfa", "node_type": "1", "metadata": {}, "hash": "dfa70486787d27016373dbaf027cd410cb9abbf2a3f7aaf98388324fb417132b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6053, "end_char_idx": 8212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:

```py
>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_opus_books_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=2,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_books["train"],
...     eval_dataset=tokenized_books["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "cfabeaa1-10f0-474c-ac3f-51684ebaecfa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c51b997-8632-494a-a2be-c1b9fbf78836", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "68a7b6eb5f325ea92841e04ea99575919c6afb04e8bb4fb599ef63024aa0279f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a77abb5c-b31f-44d4-b071-039f2e041e95", "node_type": "1", "metadata": {}, "hash": "60de0cecc43834dc4ca4d83458ffac3949d79a8ae69d8a6ff74334dc3e2572b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6709, "end_char_idx": 9715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_opus_books_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=2,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_books["train"],
...     eval_dataset=tokenized_books["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_books["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_books["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the SacreBLEU metric from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).


key: _node_content
string_value:
{"id_": "a77abb5c-b31f-44d4-b071-039f2e041e95", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cfabeaa1-10f0-474c-ac3f-51684ebaecfa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "331cd9e8db01e0215e0ee7a40cfae001cc099acf1aef0d7d3dd4b7b638a90c52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fdb6f5c8-2990-4a75-8ed1-bb1add1b9805", "node_type": "1", "metadata": {}, "hash": "8fce709c167c86bdb02defdbfc2e7ccd937efd2908b41d93774123f4918b6eb1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8214, "end_char_idx": 11445, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_books["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_books["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the SacreBLEU metric from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_opus_books_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for translation, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to translate to another language. For T5, you need to prefix your input depending on the task you're working on.


key: _node_content
string_value:
{"id_": "fdb6f5c8-2990-4a75-8ed1-bb1add1b9805", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a77abb5c-b31f-44d4-b071-039f2e041e95", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a0996d771aff0d7fa1ae9596956e8fed71515ec21a6027ed6b92c6d2ea42e9e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca117bbe-cf99-4b88-a5c7-d2dd59737b1e", "node_type": "1", "metadata": {}, "hash": "cadd133cb6e157b37e75e40f88d9a0f8eba34b1dbf7342ad37992d5fb7483578", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10430, "end_char_idx": 13442, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for translation, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to translate to another language. For T5, you need to prefix your input depending on the task you're working on. For translation from English to French, you should prefix your input as shown below:

```py
>>> text = "translate English to French: Legumes share resources with nitrogen-fixing bacteria."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for translation with your model, and pass your text to it:

```py
>>> from transformers import pipeline

# Change `xx` to the language of the input and `yy` to the language of the desired output.
# Examples: "en" for English, "fr" for French, "de" for German, "es" for Spanish, "zh" for Chinese, etc; translation_en_to_fr translates English to French
# You can view all the lists of languages here - https://huggingface.co/languages
>>> translator = pipeline("translation_xx_to_yy", model="my_awesome_opus_books_model")
>>> translator(text)
[{'translation_text': 'Legumes partagent des ressources avec des bactéries azotantes.'}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_opus_books_model")
>>> inputs = tokenizer(text, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import AutoModelForSeq2SeqLM

>>> model = AutoModelForSeq2SeqLM.from_pretrained("my_awesome_opus_books_model")
>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Les lignées partagent des ressources avec des bactéries enfixant l'azote.'


key: _node_content
string_value:
{"id_": "ca117bbe-cf99-4b88-a5c7-d2dd59737b1e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bd26c88-945a-4fa8-8440-abad11ee204d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fbfb50cc62fdeb190563bf3f4581934b1b29621c9275bc60a5ac45f6d5d91a6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fdb6f5c8-2990-4a75-8ed1-bb1add1b9805", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation/", "title": "Translation", "tokens": 3616, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5612508975fd429dc60b3ecd6e6172932df4f082e4f88b911dfe825b0d424bda", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11956, "end_char_idx": 14466, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Examples: "en" for English, "fr" for French, "de" for German, "es" for Spanish, "zh" for Chinese, etc; translation_en_to_fr translates English to French
# You can view all the lists of languages here - https://huggingface.co/languages
>>> translator = pipeline("translation_xx_to_yy", model="my_awesome_opus_books_model")
>>> translator(text)
[{'translation_text': 'Legumes partagent des ressources avec des bactéries azotantes.'}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_opus_books_model")
>>> inputs = tokenizer(text, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import AutoModelForSeq2SeqLM

>>> model = AutoModelForSeq2SeqLM.from_pretrained("my_awesome_opus_books_model")
>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Les lignées partagent des ressources avec des bactéries enfixant l'azote.'
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_opus_books_model")
>>> inputs = tokenizer(text, return_tensors="tf").input_ids
```

Use the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained("my_awesome_opus_books_model")
>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Les lugumes partagent les ressources avec des bactéries fixatrices d'azote.'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "5e2b3d11-5d68-4ef9-8c00-d394f712f8c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "335c420c-2e9d-49e8-afcb-06fad9724e11", "node_type": "1", "metadata": {}, "hash": "3311d53bd580533cd05c4af34f34169d9ecb33adad1a073090d19bfefc810391", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3090, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Document Question Answering

[[open-in-colab]]

Document Question Answering, also referred to as Document Visual Question Answering, is a task that involves providing
answers to questions posed about document images. The input to models supporting this task is typically a combination of an image and
a question, and the output is an answer expressed in natural language. These models utilize multiple modalities, including
text, the positions of words (bounding boxes), and the image itself.

This guide illustrates how to:

- Fine-tune [LayoutLMv2](../model_doc/layoutlmv2) on the [DocVQA dataset](https://huggingface.co/datasets/nielsr/docvqa_1200_examples_donut).
- Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-to-text)

</Tip>

LayoutLMv2 solves the document question-answering task by adding a question-answering head on top of the final hidden
states of the tokens, to predict the positions of the start and end tokens of the
answer. In other words, the problem is treated as extractive question answering: given the context, extract which piece
of information answers the question. The context comes from the output of an OCR engine, here it is Google's Tesseract.

Before you begin, make sure you have all the necessary libraries installed. LayoutLMv2 depends on detectron2, torchvision and tesseract.

```bash
pip install -q transformers datasets
```

```bash
pip install 'git+https://github.com/facebookresearch/detectron2.git'
pip install torchvision
```

```bash
sudo apt install tesseract-ocr
pip install -q pytesseract
```

Once you have installed all of the dependencies, restart your runtime.

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define some global variables.

```py
>>> model_checkpoint = "microsoft/layoutlmv2-base-uncased"
>>> batch_size = 4
```

## Load the data

In this guide we use a small sample of preprocessed DocVQA that you can find on 🤗 Hub. If you'd like to use the full
DocVQA dataset, you can register and download it on [DocVQA homepage](https://rrc.cvc.uab.es/?ch=17). If you do so, to
proceed with this guide check out [how to load files into a 🤗 dataset](https://huggingface.co/docs/datasets/loading#local-and-remote-files).

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("nielsr/docvqa_1200_examples")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 200
    })
})
```

As you can see, the dataset is split into train and test sets already. Take a look at a random example to familiarize
yourself with the features.


key: _node_content
string_value:
{"id_": "335c420c-2e9d-49e8-afcb-06fad9724e11", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e2b3d11-5d68-4ef9-8c00-d394f712f8c9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "990c1648c0b33566b0a1cb1972211488b7a885f8d29788b4c0ca8e39601aa197", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8cc8b5cc-704c-4b5c-b733-4fa890eb96bc", "node_type": "1", "metadata": {}, "hash": "73b6abd76c0efa6563d6d8be28363ac08bfc38810478c4a81fcaa1c24f7e7f8b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1758, "end_char_idx": 4821, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define some global variables.

```py
>>> model_checkpoint = "microsoft/layoutlmv2-base-uncased"
>>> batch_size = 4
```

## Load the data

In this guide we use a small sample of preprocessed DocVQA that you can find on 🤗 Hub. If you'd like to use the full
DocVQA dataset, you can register and download it on [DocVQA homepage](https://rrc.cvc.uab.es/?ch=17). If you do so, to
proceed with this guide check out [how to load files into a 🤗 dataset](https://huggingface.co/docs/datasets/loading#local-and-remote-files).

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("nielsr/docvqa_1200_examples")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 200
    })
})
```

As you can see, the dataset is split into train and test sets already. Take a look at a random example to familiarize
yourself with the features.

```py
>>> dataset["train"].features
```

Here's what the individual fields represent:
* `id`: the example's id
* `image`: a PIL.Image.Image object containing the document image
* `query`: the question string - natural language asked question, in several languages
* `answers`: a list of correct answers provided by human annotators
* `words` and `bounding_boxes`: the results of OCR, which we will not use here
* `answer`: an answer matched by a different model which we will not use here

Let's leave only English questions, and drop the `answer` feature which appears to contain predictions by another model.
We'll also take the first of the answers from the set provided by the annotators. Alternatively, you can randomly sample it.

```py
>>> updated_dataset = dataset.map(lambda example: {"question": example["query"]["en"]}, remove_columns=["query"])
>>> updated_dataset = updated_dataset.map(
...     lambda example: {"answer": example["answers"][0]}, remove_columns=["answer", "answers"]
... )
```

Note that the LayoutLMv2 checkpoint that we use in this guide has been trained with `max_position_embeddings = 512` (you can
find this information in the [checkpoint's `config.json` file](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json#L18)).
We can truncate the examples but to avoid the situation where the answer might be at the end of a large document and end up truncated,
here we'll remove the few examples where the embedding is likely to end up longer than 512.
If most of the documents in your dataset are long, you can implement a sliding window strategy - check out [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb) for details.


key: _node_content
string_value:
{"id_": "8cc8b5cc-704c-4b5c-b733-4fa890eb96bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "335c420c-2e9d-49e8-afcb-06fad9724e11", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fca1f1e6dc2f87cec71d4bc1ebca7b299a853865cb3b113dd7d582ae838dc4f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbca81ee-0ac2-42f6-9674-5acc96cc9181", "node_type": "1", "metadata": {}, "hash": "7c636516b9b7234d61be84065d312d4873b3300bbe82fb0cebb494b32fdf33e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3092, "end_char_idx": 6396, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> dataset["train"].features
```

Here's what the individual fields represent:
* `id`: the example's id
* `image`: a PIL.Image.Image object containing the document image
* `query`: the question string - natural language asked question, in several languages
* `answers`: a list of correct answers provided by human annotators
* `words` and `bounding_boxes`: the results of OCR, which we will not use here
* `answer`: an answer matched by a different model which we will not use here

Let's leave only English questions, and drop the `answer` feature which appears to contain predictions by another model.
We'll also take the first of the answers from the set provided by the annotators. Alternatively, you can randomly sample it.

```py
>>> updated_dataset = dataset.map(lambda example: {"question": example["query"]["en"]}, remove_columns=["query"])
>>> updated_dataset = updated_dataset.map(
...     lambda example: {"answer": example["answers"][0]}, remove_columns=["answer", "answers"]
... )
```

Note that the LayoutLMv2 checkpoint that we use in this guide has been trained with `max_position_embeddings = 512` (you can
find this information in the [checkpoint's `config.json` file](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json#L18)).
We can truncate the examples but to avoid the situation where the answer might be at the end of a large document and end up truncated,
here we'll remove the few examples where the embedding is likely to end up longer than 512.
If most of the documents in your dataset are long, you can implement a sliding window strategy - check out [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb) for details.

```py
>>> updated_dataset = updated_dataset.filter(lambda x: len(x["words"]) + len(x["question"].split()) < 512)
```

At this point let's also remove the OCR features from this dataset. These are a result of OCR for fine-tuning a different
model. They would still require some processing if we wanted to use them, as they do not match the input requirements
of the model we use in this guide. Instead, we can use the [`LayoutLMv2Processor`] on the original data for both OCR and
tokenization. This way we'll get the inputs that match model's expected input. If you want to process images manually,
check out the [`LayoutLMv2` model documentation](../model_doc/layoutlmv2) to learn what input format the model expects.

```py
>>> updated_dataset = updated_dataset.remove_columns("words")
>>> updated_dataset = updated_dataset.remove_columns("bounding_boxes")
```

Finally, the data exploration won't be complete if we don't peek at an image example.

```py
>>> updated_dataset["train"][11]["image"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/docvqa_example.jpg" alt="DocVQA Image Example"/>
 </div>

## Preprocess the data

The Document Question Answering task is a multimodal task, and you need to make sure that the inputs from each modality
are preprocessed according to the model's expectations. Let's start by loading the [`LayoutLMv2Processor`], which internally combines an image processor that can handle image data and a tokenizer that can encode text data.


key: _node_content
string_value:
{"id_": "fbca81ee-0ac2-42f6-9674-5acc96cc9181", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8cc8b5cc-704c-4b5c-b733-4fa890eb96bc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "230b393b804aa9141604ab6bd8aee9d3e84b2010c2c6e4005e3c2c2a7a5d9a3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d465fb92-7e97-47b3-b868-e99133d34153", "node_type": "1", "metadata": {}, "hash": "ef191df0b3d976823f38e66de4ec3b0ed801134b0ab52fe498f73f629556a331", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4823, "end_char_idx": 8148, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> updated_dataset = updated_dataset.filter(lambda x: len(x["words"]) + len(x["question"].split()) < 512)
```

At this point let's also remove the OCR features from this dataset. These are a result of OCR for fine-tuning a different
model. They would still require some processing if we wanted to use them, as they do not match the input requirements
of the model we use in this guide. Instead, we can use the [`LayoutLMv2Processor`] on the original data for both OCR and
tokenization. This way we'll get the inputs that match model's expected input. If you want to process images manually,
check out the [`LayoutLMv2` model documentation](../model_doc/layoutlmv2) to learn what input format the model expects.

```py
>>> updated_dataset = updated_dataset.remove_columns("words")
>>> updated_dataset = updated_dataset.remove_columns("bounding_boxes")
```

Finally, the data exploration won't be complete if we don't peek at an image example.

```py
>>> updated_dataset["train"][11]["image"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/docvqa_example.jpg" alt="DocVQA Image Example"/>
 </div>

## Preprocess the data

The Document Question Answering task is a multimodal task, and you need to make sure that the inputs from each modality
are preprocessed according to the model's expectations. Let's start by loading the [`LayoutLMv2Processor`], which internally combines an image processor that can handle image data and a tokenizer that can encode text data.

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained(model_checkpoint)
```

### Preprocessing document images

First, let's prepare the document images for the model with the help of the `image_processor` from the processor.
By default, image processor resizes the images to 224x224, makes sure they have the correct order of color channels,
applies OCR with tesseract to get words and normalized bounding boxes. In this tutorial, all of these defaults are exactly what we need.
Write a function that applies the default image processing to a batch of images and returns the results of OCR.

```py
>>> image_processor = processor.image_processor


>>> def get_ocr_words_and_boxes(examples):
...     images = [image.convert("RGB") for image in examples["image"]]
...     encoded_inputs = image_processor(images)

...     examples["image"] = encoded_inputs.pixel_values
...     examples["words"] = encoded_inputs.words
...     examples["boxes"] = encoded_inputs.boxes

...     return examples
```

To apply this preprocessing to the entire dataset in a fast way, use [`~datasets.Dataset.map`].

```py
>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)
```

### Preprocessing text data

Once we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.
This involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,
`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.

```py
>>> tokenizer = processor.tokenizer
```

On top of the preprocessing mentioned above, we also need to add the labels for the model.


key: _node_content
string_value:
{"id_": "d465fb92-7e97-47b3-b868-e99133d34153", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbca81ee-0ac2-42f6-9674-5acc96cc9181", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b673117fe433f72684f59b8ed29494e519524d9c65ec5bb9d24cecf45384b134", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e1a1fce-4dc7-41ff-b0ce-da62eedb00a9", "node_type": "1", "metadata": {}, "hash": "213a74c67dd3d49267d3a4713aed4d3b93947a49030f0b9a211a41b332a28ddc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6398, "end_char_idx": 9193, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained(model_checkpoint)
```

### Preprocessing document images

First, let's prepare the document images for the model with the help of the `image_processor` from the processor.
By default, image processor resizes the images to 224x224, makes sure they have the correct order of color channels,
applies OCR with tesseract to get words and normalized bounding boxes. In this tutorial, all of these defaults are exactly what we need.
Write a function that applies the default image processing to a batch of images and returns the results of OCR.

```py
>>> image_processor = processor.image_processor


>>> def get_ocr_words_and_boxes(examples):
...     images = [image.convert("RGB") for image in examples["image"]]
...     encoded_inputs = image_processor(images)

...     examples["image"] = encoded_inputs.pixel_values
...     examples["words"] = encoded_inputs.words
...     examples["boxes"] = encoded_inputs.boxes

...     return examples
```

To apply this preprocessing to the entire dataset in a fast way, use [`~datasets.Dataset.map`].

```py
>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)
```

### Preprocessing text data

Once we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.
This involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,
`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.

```py
>>> tokenizer = processor.tokenizer
```

On top of the preprocessing mentioned above, we also need to add the labels for the model. For `xxxForQuestionAnswering` models
in 🤗 Transformers, the labels consist of the `start_positions` and `end_positions`, indicating which token is at the
start and which token is at the end of the answer.

Let's start with that. Define a helper function that can find a sublist (the answer split into words) in a larger list (the words list).

This function will take two lists as input, `words_list` and `answer_list`. It will then iterate over the `words_list` and check
if the current word in the `words_list` (words_list[i]) is equal to the first word of answer_list (answer_list[0]) and if
the sublist of `words_list` starting from the current word and of the same length as `answer_list` is equal `to answer_list`.
If this condition is true, it means that a match has been found, and the function will record the match, its starting index (idx),
and its ending index (idx + len(answer_list) - 1). If more than one match was found, the function will return only the first one.
If no match is found, the function returns (`None`, 0, and 0).


key: _node_content
string_value:
{"id_": "8e1a1fce-4dc7-41ff-b0ce-da62eedb00a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d465fb92-7e97-47b3-b868-e99133d34153", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1d400dc3aec56a5640869aa30ced586c1a04983e4a95c56e4a37a0269d1a14f8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9f7f6dc-8688-4276-bb08-e39ce97cac56", "node_type": "1", "metadata": {}, "hash": "1be996007826f65ac69e35b683e3a4fd5423b560e465f82e8437dd414e3bf2e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7534, "end_char_idx": 10500, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)
```

### Preprocessing text data

Once we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.
This involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,
`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.

```py
>>> tokenizer = processor.tokenizer
```

On top of the preprocessing mentioned above, we also need to add the labels for the model. For `xxxForQuestionAnswering` models
in 🤗 Transformers, the labels consist of the `start_positions` and `end_positions`, indicating which token is at the
start and which token is at the end of the answer.

Let's start with that. Define a helper function that can find a sublist (the answer split into words) in a larger list (the words list).

This function will take two lists as input, `words_list` and `answer_list`. It will then iterate over the `words_list` and check
if the current word in the `words_list` (words_list[i]) is equal to the first word of answer_list (answer_list[0]) and if
the sublist of `words_list` starting from the current word and of the same length as `answer_list` is equal `to answer_list`.
If this condition is true, it means that a match has been found, and the function will record the match, its starting index (idx),
and its ending index (idx + len(answer_list) - 1). If more than one match was found, the function will return only the first one.
If no match is found, the function returns (`None`, 0, and 0).

```py
>>> def subfinder(words_list, answer_list):
...     matches = []
...     start_indices = []
...     end_indices = []
...     for idx, i in enumerate(range(len(words_list))):
...         if words_list[i] == answer_list[0] and words_list[i : i + len(answer_list)] == answer_list:
...             matches.append(answer_list)
...             start_indices.append(idx)
...             end_indices.append(idx + len(answer_list) - 1)
...     if matches:
...         return matches[0], start_indices[0], end_indices[0]
...     else:
...         return None, 0, 0
```

To illustrate how this function finds the position of the answer, let's use it on an example:

```py
>>> example = dataset_with_ocr["train"][1]
>>> words = [word.lower() for word in example["words"]]
>>> match, word_idx_start, word_idx_end = subfinder(words, example["answer"].lower().split())
>>> print("Question: ", example["question"])
>>> print("Words:", words)
>>> print("Answer: ", example["answer"])
>>> print("start_index", word_idx_start)
>>> print("end_index", word_idx_end)
Question:  Who is in  cc in this letter?
Words: ['wie', 'baw', 'brown', '&', 'williamson', 'tobacco', 'corporation', 'research', '&', 'development', 'internal', 'correspondence', 'to:', 'r.', 'h.', 'honeycutt', 'ce:', 't.f.', 'riehl', 'from:', '.', 'c.j.


key: _node_content
string_value:
{"id_": "a9f7f6dc-8688-4276-bb08-e39ce97cac56", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e1a1fce-4dc7-41ff-b0ce-da62eedb00a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7d82c9b96c8903299b9d350890c15b2798692b3e7d37f20712154d51345f9cc8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "564e0bea-0e9b-4de8-ad94-d062cd3d7843", "node_type": "1", "metadata": {}, "hash": "ea9c2ad6fe327da3d8086b56381da8f049b2d2abf9ddd19de0e2b8824ebccb45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9052, "end_char_idx": 11606, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If more than one match was found, the function will return only the first one.
If no match is found, the function returns (`None`, 0, and 0).

```py
>>> def subfinder(words_list, answer_list):
...     matches = []
...     start_indices = []
...     end_indices = []
...     for idx, i in enumerate(range(len(words_list))):
...         if words_list[i] == answer_list[0] and words_list[i : i + len(answer_list)] == answer_list:
...             matches.append(answer_list)
...             start_indices.append(idx)
...             end_indices.append(idx + len(answer_list) - 1)
...     if matches:
...         return matches[0], start_indices[0], end_indices[0]
...     else:
...         return None, 0, 0
```

To illustrate how this function finds the position of the answer, let's use it on an example:

```py
>>> example = dataset_with_ocr["train"][1]
>>> words = [word.lower() for word in example["words"]]
>>> match, word_idx_start, word_idx_end = subfinder(words, example["answer"].lower().split())
>>> print("Question: ", example["question"])
>>> print("Words:", words)
>>> print("Answer: ", example["answer"])
>>> print("start_index", word_idx_start)
>>> print("end_index", word_idx_end)
Question:  Who is in  cc in this letter?
Words: ['wie', 'baw', 'brown', '&', 'williamson', 'tobacco', 'corporation', 'research', '&', 'development', 'internal', 'correspondence', 'to:', 'r.', 'h.', 'honeycutt', 'ce:', 't.f.', 'riehl', 'from:', '.', 'c.j.', 'cook', 'date:', 'may', '8,', '1995', 'subject:', 'review', 'of', 'existing', 'brainstorming', 'ideas/483', 'the', 'major', 'function', 'of', 'the', 'product', 'innovation', 'graup', 'is', 'to', 'develop', 'marketable', 'nove!', 'products', 'that', 'would', 'be', 'profitable', 'to', 'manufacture', 'and', 'sell.', 'novel', 'is', 'defined', 'as:', 'of', 'a', 'new', 'kind,', 'or', 'different', 'from', 'anything', 'seen', 'or', 'known', 'before.', 'innovation', 'is', 'defined', 'as:', 'something', 'new', 'or', 'different', 'introduced;', 'act', 'of', 'innovating;', 'introduction', 'of', 'new', 'things', 'or', 'methods.', 'the', 'products', 'may', 'incorporate', 'the', 'latest', 'technologies,', 'materials', 'and', 'know-how', 'available', 'to', 'give', 'then', 'a', 'unique', 'taste', 'or', 'look.', 'the', 'first', 'task', 'of', 'the', 'product', 'innovation', 'group', 'was', 'to', 'assemble,', 'review', 'and', 'categorize', 'a', 'list', 'of', 'existing', 'brainstorming', 'ideas.', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.


key: _node_content
string_value:
{"id_": "564e0bea-0e9b-4de8-ad94-d062cd3d7843", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9f7f6dc-8688-4276-bb08-e39ce97cac56", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5bfa6edb0126c2e3b02a4b9b4db87f985c719d5e7a509576a47f0ee34b324e06", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "782c468d-a265-4c29-b45f-4c6cbcb6acb6", "node_type": "1", "metadata": {}, "hash": "6c23259061dd04b077ba3316885b24d1c5eb20b10475aadfe7b2a78f363e73f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10492, "end_char_idx": 12569, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', 'c.j.', 'cook', 'date:', 'may', '8,', '1995', 'subject:', 'review', 'of', 'existing', 'brainstorming', 'ideas/483', 'the', 'major', 'function', 'of', 'the', 'product', 'innovation', 'graup', 'is', 'to', 'develop', 'marketable', 'nove!', 'products', 'that', 'would', 'be', 'profitable', 'to', 'manufacture', 'and', 'sell.', 'novel', 'is', 'defined', 'as:', 'of', 'a', 'new', 'kind,', 'or', 'different', 'from', 'anything', 'seen', 'or', 'known', 'before.', 'innovation', 'is', 'defined', 'as:', 'something', 'new', 'or', 'different', 'introduced;', 'act', 'of', 'innovating;', 'introduction', 'of', 'new', 'things', 'or', 'methods.', 'the', 'products', 'may', 'incorporate', 'the', 'latest', 'technologies,', 'materials', 'and', 'know-how', 'available', 'to', 'give', 'then', 'a', 'unique', 'taste', 'or', 'look.', 'the', 'first', 'task', 'of', 'the', 'product', 'innovation', 'group', 'was', 'to', 'assemble,', 'review', 'and', 'categorize', 'a', 'list', 'of', 'existing', 'brainstorming', 'ideas.', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.', 'these', 'categories', 'are', 'used', 'for', 'novel', 'products', 'that', 'may', 'differ', 'from', 'a', 'visual', 'and/or', 'taste/aroma', 'point', 'of', 'view', 'compared', 'to', 'canventional', 'cigarettes.', 'other', 'categories', 'include', 'a', 'combination', 'of', 'the', 'above,', 'filters,', 'packaging', 'and', 'brand', 'extensions.', 'appearance', 'this', 'category', 'is', 'used', 'for', 'novel', 'cigarette', 'constructions', 'that', 'yield', 'visually', 'different', 'products', 'with', 'minimal', 'changes', 'in', 'smoke', 'chemistry', 'two', 'cigarettes', 'in', 'cne.', 'emulti-plug', 'te', 'build', 'yaur', 'awn', 'cigarette.', 'eswitchable', 'menthol', 'or', 'non', 'menthol', 'cigarette.', '*cigarettes', 'with', 'interspaced', 'perforations', 'to', 'enable', 'smoker', 'to', 'separate', 'unburned', 'section', 'for', 'future', 'smoking.', '«short', 'cigarette,', 'tobacco', 'section', '30', 'mm.', '«extremely', 'fast', 'buming', 'cigarette.


key: _node_content
string_value:
{"id_": "782c468d-a265-4c29-b45f-4c6cbcb6acb6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "564e0bea-0e9b-4de8-ad94-d062cd3d7843", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "65abc655028d641dd9fcbdc1acf731f44210d941802a7b109142352f529c5e38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2875ea77-b8d7-4a36-9671-e2cb01dad95f", "node_type": "1", "metadata": {}, "hash": "253e53b788e21308ff0d22b7e8776240d5b82a2929f38ffb5874994a70b8906d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11492, "end_char_idx": 13724, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.', 'these', 'categories', 'are', 'used', 'for', 'novel', 'products', 'that', 'may', 'differ', 'from', 'a', 'visual', 'and/or', 'taste/aroma', 'point', 'of', 'view', 'compared', 'to', 'canventional', 'cigarettes.', 'other', 'categories', 'include', 'a', 'combination', 'of', 'the', 'above,', 'filters,', 'packaging', 'and', 'brand', 'extensions.', 'appearance', 'this', 'category', 'is', 'used', 'for', 'novel', 'cigarette', 'constructions', 'that', 'yield', 'visually', 'different', 'products', 'with', 'minimal', 'changes', 'in', 'smoke', 'chemistry', 'two', 'cigarettes', 'in', 'cne.', 'emulti-plug', 'te', 'build', 'yaur', 'awn', 'cigarette.', 'eswitchable', 'menthol', 'or', 'non', 'menthol', 'cigarette.', '*cigarettes', 'with', 'interspaced', 'perforations', 'to', 'enable', 'smoker', 'to', 'separate', 'unburned', 'section', 'for', 'future', 'smoking.', '«short', 'cigarette,', 'tobacco', 'section', '30', 'mm.', '«extremely', 'fast', 'buming', 'cigarette.', '«novel', 'cigarette', 'constructions', 'that', 'permit', 'a', 'significant', 'reduction', 'iretobacco', 'weight', 'while', 'maintaining', 'smoking', 'mechanics', 'and', 'visual', 'characteristics.', 'higher', 'basis', 'weight', 'paper:', 'potential', 'reduction', 'in', 'tobacco', 'weight.', '«more', 'rigid', 'tobacco', 'column;', 'stiffing', 'agent', 'for', 'tobacco;', 'e.g.', 'starch', '*colored', 'tow', 'and', 'cigarette', 'papers;', 'seasonal', 'promotions,', 'e.g.', 'pastel', 'colored', 'cigarettes', 'for', 'easter', 'or', 'in', 'an', 'ebony', 'and', 'ivory', 'brand', 'containing', 'a', 'mixture', 'of', 'all', 'black', '(black', 'paper', 'and', 'tow)', 'and', 'ail', 'white', 'cigarettes.', '499150498']
Answer:  T.F. Riehl
start_index 17
end_index 18
```

Once examples are encoded, however, they will look like this:

```py
>>> encoding = tokenizer(example["question"], example["words"], example["boxes"])
>>> tokenizer.decode(encoding["input_ids"])
[CLS] who is in cc in this letter? [SEP] wie baw brown & williamson tobacco corporation research & development ...
```

We'll need to find the position of the answer in the encoded input.


key: _node_content
string_value:
{"id_": "2875ea77-b8d7-4a36-9671-e2cb01dad95f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "782c468d-a265-4c29-b45f-4c6cbcb6acb6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5713d0f2bccea569824eabc16a5c6851f7977858cb86dc3e3b199b0a342dec4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "386dd946-2852-4222-8eac-cf1fb47c3abe", "node_type": "1", "metadata": {}, "hash": "54858536e85b1941144a1b1dd7af4ca00dafcc251b84a08e6b6410c9b52b4338", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', '«extremely', 'fast', 'buming', 'cigarette.', '«novel', 'cigarette', 'constructions', 'that', 'permit', 'a', 'significant', 'reduction', 'iretobacco', 'weight', 'while', 'maintaining', 'smoking', 'mechanics', 'and', 'visual', 'characteristics.', 'higher', 'basis', 'weight', 'paper:', 'potential', 'reduction', 'in', 'tobacco', 'weight.', '«more', 'rigid', 'tobacco', 'column;', 'stiffing', 'agent', 'for', 'tobacco;', 'e.g.', 'starch', '*colored', 'tow', 'and', 'cigarette', 'papers;', 'seasonal', 'promotions,', 'e.g.', 'pastel', 'colored', 'cigarettes', 'for', 'easter', 'or', 'in', 'an', 'ebony', 'and', 'ivory', 'brand', 'containing', 'a', 'mixture', 'of', 'all', 'black', '(black', 'paper', 'and', 'tow)', 'and', 'ail', 'white', 'cigarettes.', '499150498']
Answer:  T.F. Riehl
start_index 17
end_index 18
```

Once examples are encoded, however, they will look like this:

```py
>>> encoding = tokenizer(example["question"], example["words"], example["boxes"])
>>> tokenizer.decode(encoding["input_ids"])
[CLS] who is in cc in this letter? [SEP] wie baw brown & williamson tobacco corporation research & development ...
```

We'll need to find the position of the answer in the encoded input.
* `token_type_ids` tells us which tokens are part of the question, and which ones are part of the document's words.
* `tokenizer.cls_token_id` will help find the special token at the beginning of the input.
* `word_ids` will help match the answer found in the original `words` to the same answer in the full encoded input and determine
the start/end position of the answer in the encoded input.

With that in mind, let's create a function to encode a batch of examples in the dataset:

```py
>>> def encode_dataset(examples, max_length=512):
.     questions = examples["question"]
.     words = examples["words"]
.     boxes = examples["boxes"]
.     answers = examples["answer"]

.     # encode the batch of examples and initialize the start_positions and end_positions
.     encoding = tokenizer(questions, words, boxes, max_length=max_length, padding="max_length", truncation=True)
.     start_positions = []
.     end_positions = []

.     # loop through the examples in the batch
.     for i in range(len(questions)):
.         cls_index = encoding["input_ids"][i].index(tokenizer.cls_token_id)

.         # find the position of the answer in example's words
.         words_example = [word.lower() for word in words[i]]
.         answer = answers[i]
.         match, word_idx_start, word_idx_end = subfinder(words_example, answer.lower().split())

.         if match:
.             # if match is found, use `token_type_ids` to find where words start in the encoding
.             token_type_ids = encoding["token_type_ids"][i]
.             token_start_index = 0
.             while token_type_ids[token_start_index] != 1:
.                 token_start_index += 1

.


key: _node_content
string_value:
{"id_": "386dd946-2852-4222-8eac-cf1fb47c3abe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2875ea77-b8d7-4a36-9671-e2cb01dad95f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "67368509a738e8dcf0d1d7d1dccf7f66dc44682ba8703f23963faf04b6d5e0df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d521427-d96a-4219-ab05-fe0d335dca0e", "node_type": "1", "metadata": {}, "hash": "c927672daa6187c26f9de6e8cbcd69667d0de0f3c3bdca2b4b5c19d3cfa5e200", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* `token_type_ids` tells us which tokens are part of the question, and which ones are part of the document's words.
* `tokenizer.cls_token_id` will help find the special token at the beginning of the input.
* `word_ids` will help match the answer found in the original `words` to the same answer in the full encoded input and determine
the start/end position of the answer in the encoded input.

With that in mind, let's create a function to encode a batch of examples in the dataset:

```py
>>> def encode_dataset(examples, max_length=512):
.     questions = examples["question"]
.     words = examples["words"]
.     boxes = examples["boxes"]
.     answers = examples["answer"]

.     # encode the batch of examples and initialize the start_positions and end_positions
.     encoding = tokenizer(questions, words, boxes, max_length=max_length, padding="max_length", truncation=True)
.     start_positions = []
.     end_positions = []

.     # loop through the examples in the batch
.     for i in range(len(questions)):
.         cls_index = encoding["input_ids"][i].index(tokenizer.cls_token_id)

.         # find the position of the answer in example's words
.         words_example = [word.lower() for word in words[i]]
.         answer = answers[i]
.         match, word_idx_start, word_idx_end = subfinder(words_example, answer.lower().split())

.         if match:
.             # if match is found, use `token_type_ids` to find where words start in the encoding
.             token_type_ids = encoding["token_type_ids"][i]
.             token_start_index = 0
.             while token_type_ids[token_start_index] != 1:
.                 token_start_index += 1

.             token_end_index = len(encoding["input_ids"][i]) - 1
.             while token_type_ids[token_end_index] != 1:
.                 token_end_index -= 1

.             word_ids = encoding.word_ids(i)[token_start_index : token_end_index + 1]
.             start_position = cls_index
.             end_position = cls_index

.             # loop over word_ids and increase `token_start_index` until it matches the answer position in words
.             # once it matches, save the `token_start_index` as the `start_position` of the answer in the encoding
.             for id in word_ids:
.                 if id == word_idx_start:
.                     start_position = token_start_index
.                 else:
.                     token_start_index += 1

.             # similarly loop over `word_ids` starting from the end to find the `end_position` of the answer
.             for id in word_ids[::-1]:
.                 if id == word_idx_end:
.                     end_position = token_end_index
.                 else:
.                     token_end_index -= 1

.             start_positions.append(start_position)
.             end_positions.append(end_position)

.         else:
.             start_positions.append(cls_index)
.             end_positions.append(cls_index)

.     encoding["image"] = examples["image"]
.     encoding["start_positions"] = start_positions
.     encoding["end_positions"] = end_positions

.     return encoding
```

Now that we have this preprocessing function, we can encode the entire dataset:

```py
>>> encoded_train_dataset = dataset_with_ocr["train"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["train"].column_names
. )
>>> encoded_test_dataset = dataset_with_ocr["test"].map(
.     encode_dataset,


key: _node_content
string_value:
{"id_": "8d521427-d96a-4219-ab05-fe0d335dca0e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "386dd946-2852-4222-8eac-cf1fb47c3abe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7478631999cccc2297b6af2e13dce25f6d7e34a2269638a28ccfa85c5cd8f59c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c4241d5-1820-4fef-8c65-ec71d173972b", "node_type": "1", "metadata": {}, "hash": "5aaf0084b882778a32ed68bbac54f34aee5df2b88aa330d7e5bfd78ae2a700e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
token_end_index = len(encoding["input_ids"][i]) - 1
.             while token_type_ids[token_end_index] != 1:
.                 token_end_index -= 1

.             word_ids = encoding.word_ids(i)[token_start_index : token_end_index + 1]
.             start_position = cls_index
.             end_position = cls_index

.             # loop over word_ids and increase `token_start_index` until it matches the answer position in words
.             # once it matches, save the `token_start_index` as the `start_position` of the answer in the encoding
.             for id in word_ids:
.                 if id == word_idx_start:
.                     start_position = token_start_index
.                 else:
.                     token_start_index += 1

.             # similarly loop over `word_ids` starting from the end to find the `end_position` of the answer
.             for id in word_ids[::-1]:
.                 if id == word_idx_end:
.                     end_position = token_end_index
.                 else:
.                     token_end_index -= 1

.             start_positions.append(start_position)
.             end_positions.append(end_position)

.         else:
.             start_positions.append(cls_index)
.             end_positions.append(cls_index)

.     encoding["image"] = examples["image"]
.     encoding["start_positions"] = start_positions
.     encoding["end_positions"] = end_positions

.     return encoding
```

Now that we have this preprocessing function, we can encode the entire dataset:

```py
>>> encoded_train_dataset = dataset_with_ocr["train"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["train"].column_names
. )
>>> encoded_test_dataset = dataset_with_ocr["test"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["test"].column_names
. )
```

Let's check what the features of the encoded dataset look like:

```py
>>> encoded_train_dataset.features
{'image': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='uint8', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'bbox': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),
 'start_positions': Value(dtype='int64', id=None),
 'end_positions': Value(dtype='int64', id=None)}
```

## Evaluation

Evaluation for document question answering requires a significant amount of postprocessing. To avoid taking up too much
of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so
you're not completely in the dark about your model's performance. Extractive question answering is typically evaluated using F1/exact match.
If you'd like to implement it yourself, check out the [Question Answering chapter](https://huggingface.co/course/chapter7/7?fw=pt#postprocessing)
of the Hugging Face course for inspiration.

## Train

Congratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.


key: _node_content
string_value:
{"id_": "4c4241d5-1820-4fef-8c65-ec71d173972b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d521427-d96a-4219-ab05-fe0d335dca0e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1b8e6cb2af1ad2fb4936ece7aef1e85a13922c52f03fe0d5b2c0624274069960", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7120e54b-bbfa-4303-9fac-9338b7fe4814", "node_type": "1", "metadata": {}, "hash": "2e1b30a09304fb5cc22c2f9e4cdf2223944f6f2965c3355c478a475343182708", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
column_names
. )
>>> encoded_test_dataset = dataset_with_ocr["test"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["test"].column_names
. )
```

Let's check what the features of the encoded dataset look like:

```py
>>> encoded_train_dataset.features
{'image': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='uint8', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'bbox': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),
 'start_positions': Value(dtype='int64', id=None),
 'end_positions': Value(dtype='int64', id=None)}
```

## Evaluation

Evaluation for document question answering requires a significant amount of postprocessing. To avoid taking up too much
of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so
you're not completely in the dark about your model's performance. Extractive question answering is typically evaluated using F1/exact match.
If you'd like to implement it yourself, check out the [Question Answering chapter](https://huggingface.co/course/chapter7/7?fw=pt#postprocessing)
of the Hugging Face course for inspiration.

## Train

Congratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.
Training involves the following steps:
* Load the model with [`AutoModelForDocumentQuestionAnswering`] using the same checkpoint as in the preprocessing.
* Define your training hyperparameters in [`TrainingArguments`].
* Define a function to batch examples together, here the [`DefaultDataCollator`] will do just fine
* Pass the training arguments to [`Trainer`] along with the model, dataset, and data collator.
* Call [`~Trainer.train`] to finetune your model.

```py
>>> from transformers import AutoModelForDocumentQuestionAnswering

>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(model_checkpoint)
```

In the [`TrainingArguments`] use `output_dir` to specify where to save your model, and configure hyperparameters as you see fit.
If you wish to share your model with the community, set `push_to_hub` to `True` (you must be signed in to Hugging Face to upload your model).
In this case the `output_dir` will also be the name of the repo where your model checkpoint will be pushed.

```py
>>> from transformers import TrainingArguments

>>> # REPLACE THIS WITH YOUR REPO ID
>>> repo_id = "MariaK/layoutlmv2-base-uncased_finetuned_docvqa"

>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...     logging_steps=50,
...     eval_strategy="steps",
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
```

Define a simple data collator to batch examples together.


key: _node_content
string_value:
{"id_": "7120e54b-bbfa-4303-9fac-9338b7fe4814", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4c4241d5-1820-4fef-8c65-ec71d173972b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "521bf0d5d0cb79e5510ac5d2709295e83686e2499f92cc7959282667ba5af73f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47e91787-0e74-47f3-9e77-dd6f834c20a4", "node_type": "1", "metadata": {}, "hash": "22529ad00d60bfd6d24635cffbca33838f8fe8329dcc8e26d21e8de59a4d3278", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18719, "end_char_idx": 21662, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

Congratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.
Training involves the following steps:
* Load the model with [`AutoModelForDocumentQuestionAnswering`] using the same checkpoint as in the preprocessing.
* Define your training hyperparameters in [`TrainingArguments`].
* Define a function to batch examples together, here the [`DefaultDataCollator`] will do just fine
* Pass the training arguments to [`Trainer`] along with the model, dataset, and data collator.
* Call [`~Trainer.train`] to finetune your model.

```py
>>> from transformers import AutoModelForDocumentQuestionAnswering

>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(model_checkpoint)
```

In the [`TrainingArguments`] use `output_dir` to specify where to save your model, and configure hyperparameters as you see fit.
If you wish to share your model with the community, set `push_to_hub` to `True` (you must be signed in to Hugging Face to upload your model).
In this case the `output_dir` will also be the name of the repo where your model checkpoint will be pushed.

```py
>>> from transformers import TrainingArguments

>>> # REPLACE THIS WITH YOUR REPO ID
>>> repo_id = "MariaK/layoutlmv2-base-uncased_finetuned_docvqa"

>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...     logging_steps=50,
...     eval_strategy="steps",
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
```

Define a simple data collator to batch examples together.

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

Finally, bring everything together, and call [`~Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=encoded_train_dataset,
...     eval_dataset=encoded_test_dataset,
...     tokenizer=processor,
... )

>>> trainer.train()
```

To add the final model to 🤗 Hub, create a model card and call `push_to_hub`:

```py
>>> trainer.create_model_card()
>>> trainer.push_to_hub()
```

## Inference

Now that you have finetuned a LayoutLMv2 model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your finetuned model for inference is to use it in a [`Pipeline`].

Let's take an example:
```py
>>> example = dataset["test"][2]
>>> question = example["query"]["en"]
>>> image = example["image"]
>>> print(question)
>>> print(example["answers"])
'Who is ‘presiding’ TRRF GENERAL SESSION (PART 1)?'
['TRRF Vice President', 'lee a. waller']
```

Next, instantiate a pipeline for
document question answering with your model, and pass the image + question combination to it.


key: _node_content
string_value:
{"id_": "47e91787-0e74-47f3-9e77-dd6f834c20a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8d37737f-d84b-4dc8-9e2a-4e6e5649d2d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d4dd46cd4e8d38ecd1852b40ad7de209464d72ea574f3550e26429c5dc5de935", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7120e54b-bbfa-4303-9fac-9338b7fe4814", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering/", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3d89b85f469260e6e3cf85d0e7933ace070746412c92da370e46dfb0f5974cda", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20423, "end_char_idx": 23379, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

Finally, bring everything together, and call [`~Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=encoded_train_dataset,
...     eval_dataset=encoded_test_dataset,
...     tokenizer=processor,
... )

>>> trainer.train()
```

To add the final model to 🤗 Hub, create a model card and call `push_to_hub`:

```py
>>> trainer.create_model_card()
>>> trainer.push_to_hub()
```

## Inference

Now that you have finetuned a LayoutLMv2 model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your finetuned model for inference is to use it in a [`Pipeline`].

Let's take an example:
```py
>>> example = dataset["test"][2]
>>> question = example["query"]["en"]
>>> image = example["image"]
>>> print(question)
>>> print(example["answers"])
'Who is ‘presiding’ TRRF GENERAL SESSION (PART 1)?'
['TRRF Vice President', 'lee a. waller']
```

Next, instantiate a pipeline for
document question answering with your model, and pass the image + question combination to it.

```py
>>> from transformers import pipeline

>>> qa_pipeline = pipeline("document-question-answering", model="MariaK/layoutlmv2-base-uncased_finetuned_docvqa")
>>> qa_pipeline(image, question)
[{'score': 0.9949808120727539,
  'answer': 'Lee A. Waller',
  'start': 55,
  'end': 57}]
```

You can also manually replicate the results of the pipeline if you'd like:
1. Take an image and a question, prepare them for the model using the processor from your model.
2. Forward the result or preprocessing through the model.
3. The model returns `start_logits` and `end_logits`, which indicate which token is at the start of the answer and
which token is at the end of the answer. Both have shape (batch_size, sequence_length).
4. Take an argmax on the last dimension of both the `start_logits` and `end_logits` to get the predicted `start_idx` and `end_idx`.
5. Decode the answer with the tokenizer.

```py
>>> import torch
>>> from transformers import AutoProcessor
>>> from transformers import AutoModelForDocumentQuestionAnswering

>>> processor = AutoProcessor.from_pretrained("MariaK/layoutlmv2-base-uncased_finetuned_docvqa")
>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained("MariaK/layoutlmv2-base-uncased_finetuned_docvqa")

>>> with torch.no_grad():
...     encoding = processor(image.convert("RGB"), question, return_tensors="pt")
...     outputs = model(**encoding)
...     start_logits = outputs.start_logits
...     end_logits = outputs.end_logits
...     predicted_start_idx = start_logits.argmax(-1).item()
...     predicted_end_idx = end_logits.argmax(-1).item()

>>> processor.tokenizer.decode(encoding.input_ids.squeeze()[predicted_start_idx : predicted_end_idx + 1])
'lee a. waller'
```


key: _node_content
string_value:
{"id_": "b4fe5c4a-ec9e-4017-9610-1631b4257297", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c6a37581-d9dc-42ef-9cf0-0ef0726d30a2", "node_type": "1", "metadata": {}, "hash": "23212da2355ce05c4a828ec946ef7f3ba3f0a16cbfa6a29ddbe3703169d54c2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3066, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image tasks with IDEFICS

[[open-in-colab]]

While individual tasks can be tackled by fine-tuning specialized models, an alternative approach 
that has recently emerged and gained popularity is to use large models for a diverse set of tasks without fine-tuning. 
For instance, large language models can handle such NLP tasks as summarization, translation, classification, and more. 
This approach is no longer limited to a single modality, such as text, and in this guide, we will illustrate how you can 
solve image-text tasks with a large multimodal model called IDEFICS. 

[IDEFICS](../model_doc/idefics) is an open-access vision and language model based on [Flamingo](https://huggingface.co/papers/2204.14198), 
a state-of-the-art visual language model initially developed by DeepMind. The model accepts arbitrary sequences of image 
and text inputs and generates coherent text as output. It can answer questions about images, describe visual content, 
create stories grounded in multiple images, and so on. IDEFICS comes in two variants - [80 billion parameters](https://huggingface.co/HuggingFaceM4/idefics-80b) 
and [9 billion parameters](https://huggingface.co/HuggingFaceM4/idefics-9b), both of which are available on the 🤗 Hub. For each variant, you can also find fine-tuned instructed 
versions of the model adapted for conversational use cases.

This model is exceptionally versatile and can be used for a wide range of image and multimodal tasks. However, 
being a large model means it requires significant computational resources and infrastructure. It is up to you to decide whether 
this approach suits your use case better than fine-tuning specialized models for each individual task. 

In this guide, you'll learn how to: 
- [Load IDEFICS](#loading-the-model) and [load the quantized version of the model](#quantized-model)
- Use IDEFICS for: 
  - [Image captioning](#image-captioning)
  - [Prompted image captioning](#prompted-image-captioning)
  - [Few-shot prompting](#few-shot-prompting)
  - [Visual question answering](#visual-question-answering)
  - [Image classification](#image-classification)
  - [Image-guided text generation](#image-guided-text-generation)
- [Run inference in batch mode](#running-inference-in-batch-mode)
- [Run IDEFICS instruct for conversational use](#idefics-instruct-for-conversational-use)

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q bitsandbytes sentencepiece accelerate transformers
```

<Tip>
To run the following examples with a non-quantized version of the model checkpoint you will need at least 20GB of GPU memory.
</Tip>

## Loading the model

Let's start by loading the model's 9 billion parameters checkpoint: 

```py
>>> checkpoint = "HuggingFaceM4/idefics-9b"
```

Just like for other Transformers models, you need to load a processor and the model itself from the checkpoint. 
The IDEFICS processor wraps a [`LlamaTokenizer`] and IDEFICS image processor into a single processor to take care of 
preparing text and image inputs for the model.


key: _node_content
string_value:
{"id_": "c6a37581-d9dc-42ef-9cf0-0ef0726d30a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b4fe5c4a-ec9e-4017-9610-1631b4257297", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "338bd6ff76f5e5f10831b079625ed4bdee73b0a65843f6dd7bb7a70f7c2add52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b244fb41-8441-48fd-947e-72687b51d7d1", "node_type": "1", "metadata": {}, "hash": "0e3a1787316ae602c29724caaa88031d9c8349d04c11dd73e2d51ae20a66c287", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1462, "end_char_idx": 4670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, 
being a large model means it requires significant computational resources and infrastructure. It is up to you to decide whether 
this approach suits your use case better than fine-tuning specialized models for each individual task. 

In this guide, you'll learn how to: 
- [Load IDEFICS](#loading-the-model) and [load the quantized version of the model](#quantized-model)
- Use IDEFICS for: 
  - [Image captioning](#image-captioning)
  - [Prompted image captioning](#prompted-image-captioning)
  - [Few-shot prompting](#few-shot-prompting)
  - [Visual question answering](#visual-question-answering)
  - [Image classification](#image-classification)
  - [Image-guided text generation](#image-guided-text-generation)
- [Run inference in batch mode](#running-inference-in-batch-mode)
- [Run IDEFICS instruct for conversational use](#idefics-instruct-for-conversational-use)

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q bitsandbytes sentencepiece accelerate transformers
```

<Tip>
To run the following examples with a non-quantized version of the model checkpoint you will need at least 20GB of GPU memory.
</Tip>

## Loading the model

Let's start by loading the model's 9 billion parameters checkpoint: 

```py
>>> checkpoint = "HuggingFaceM4/idefics-9b"
```

Just like for other Transformers models, you need to load a processor and the model itself from the checkpoint. 
The IDEFICS processor wraps a [`LlamaTokenizer`] and IDEFICS image processor into a single processor to take care of 
preparing text and image inputs for the model.

```py
>>> import torch

>>> from transformers import IdeficsForVisionText2Text, AutoProcessor

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Setting `device_map` to `"auto"` will automatically determine how to load and store the model weights in the most optimized 
manner given existing devices.

### Quantized model

If high-memory GPU availability is an issue, you can load the quantized version of the model. To load the model and the 
processor in 4bit precision, pass a `BitsAndBytesConfig` to the `from_pretrained` method and the model will be compressed 
on the fly while loading.

```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig

>>> quantization_config = BitsAndBytesConfig(
...     load_in_4bit=True,
...     bnb_4bit_compute_dtype=torch.float16,
... )

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(
...     checkpoint,
...     quantization_config=quantization_config,
...     device_map="auto"
... )
```

Now that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.

## Image captioning
Image captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired 
people navigate through different situations, for instance, explore image content online. 

To illustrate the task, get an image to be captioned, e.g.


key: _node_content
string_value:
{"id_": "b244fb41-8441-48fd-947e-72687b51d7d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c6a37581-d9dc-42ef-9cf0-0ef0726d30a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8806a42e66b81bd8556f9b1e66fff005bffa18313028fa3f48aa51d419074049", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "81a42f10-94c3-4d9c-9660-08c815968d91", "node_type": "1", "metadata": {}, "hash": "6314744aa142dec6c5699d8efa4a2c87b252929610d77a0eb3e9fa5fd71c0a4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3068, "end_char_idx": 5366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> import torch

>>> from transformers import IdeficsForVisionText2Text, AutoProcessor

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Setting `device_map` to `"auto"` will automatically determine how to load and store the model weights in the most optimized 
manner given existing devices.

### Quantized model

If high-memory GPU availability is an issue, you can load the quantized version of the model. To load the model and the 
processor in 4bit precision, pass a `BitsAndBytesConfig` to the `from_pretrained` method and the model will be compressed 
on the fly while loading.

```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig

>>> quantization_config = BitsAndBytesConfig(
...     load_in_4bit=True,
...     bnb_4bit_compute_dtype=torch.float16,
... )

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(
...     checkpoint,
...     quantization_config=quantization_config,
...     device_map="auto"
... )
```

Now that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.

## Image captioning
Image captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired 
people navigate through different situations, for instance, explore image content online. 

To illustrate the task, get an image to be captioned, e.g.:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-im-captioning.jpg" alt="Image of a puppy in a flower bed"/>
</div>

Photo by [Hendo Wang](https://unsplash.com/@hendoo). 

IDEFICS accepts text and image prompts. However, to caption an image, you do not have to provide a text prompt to the 
model, only the preprocessed input image. Without a text prompt, the model will start generating text from the 
BOS (beginning-of-sequence) token thus creating a caption.

As image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.


key: _node_content
string_value:
{"id_": "81a42f10-94c3-4d9c-9660-08c815968d91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b244fb41-8441-48fd-947e-72687b51d7d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8b1f1f358528fbc0627919b354858731ff84c2cb8001b2bbae2925336f6da5f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "371a5796-9927-447e-9287-f0cd7428e855", "node_type": "1", "metadata": {}, "hash": "65ea3011d628dee36c517d306deb9deeaf604bef2ac68d605ca3af5917cb66a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3789, "end_char_idx": 6581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig

>>> quantization_config = BitsAndBytesConfig(
...     load_in_4bit=True,
...     bnb_4bit_compute_dtype=torch.float16,
... )

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(
...     checkpoint,
...     quantization_config=quantization_config,
...     device_map="auto"
... )
```

Now that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.

## Image captioning
Image captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired 
people navigate through different situations, for instance, explore image content online. 

To illustrate the task, get an image to be captioned, e.g.:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-im-captioning.jpg" alt="Image of a puppy in a flower bed"/>
</div>

Photo by [Hendo Wang](https://unsplash.com/@hendoo). 

IDEFICS accepts text and image prompts. However, to caption an image, you do not have to provide a text prompt to the 
model, only the preprocessed input image. Without a text prompt, the model will start generating text from the 
BOS (beginning-of-sequence) token thus creating a caption.

As image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3542&q=80",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
A puppy in a flower bed
```

<Tip>

It is a good idea to include the `bad_words_ids` in the call to `generate` to avoid errors arising when increasing 
the `max_new_tokens`: the model will want to generate a new `<image>` or `<fake_token_around_image>` token when there 
is no image being generated by the model.
You can set it on-the-fly as in this guide, or store in the `GenerationConfig` as described in the [Text generation strategies](../generation_strategies) guide.
</Tip>

## Prompted image captioning

You can extend image captioning by providing a text prompt, which the model will continue given the image.


key: _node_content
string_value:
{"id_": "371a5796-9927-447e-9287-f0cd7428e855", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "81a42f10-94c3-4d9c-9660-08c815968d91", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "71a28b9ea5facdb8e7c6783a3fed9878dabfe39090d705fbe112c62f59fe0b47", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "917f8815-8b3c-4a15-8e0b-f775425bbf3c", "node_type": "1", "metadata": {}, "hash": "654548ee4c21d47b1c2af7c733295e93ef9e27c3a9323b1a09b738509de0df7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5111, "end_char_idx": 7991, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Without a text prompt, the model will start generating text from the 
BOS (beginning-of-sequence) token thus creating a caption.

As image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3542&q=80",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
A puppy in a flower bed
```

<Tip>

It is a good idea to include the `bad_words_ids` in the call to `generate` to avoid errors arising when increasing 
the `max_new_tokens`: the model will want to generate a new `<image>` or `<fake_token_around_image>` token when there 
is no image being generated by the model.
You can set it on-the-fly as in this guide, or store in the `GenerationConfig` as described in the [Text generation strategies](../generation_strategies) guide.
</Tip>

## Prompted image captioning

You can extend image captioning by providing a text prompt, which the model will continue given the image. Let's take 
another image to illustrate:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-prompted-im-captioning.jpg" alt="Image of the Eiffel Tower at night"/>
</div>

Photo by [Denys Nevozhai](https://unsplash.com/@dnevozhai).
   
Textual and image prompts can be passed to the model's processor as a single list to create appropriate inputs.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...     "This is an image of ",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
This is an image of the Eiffel Tower in Paris, France.
```

## Few-shot prompting

While IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with 
other restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.


key: _node_content
string_value:
{"id_": "917f8815-8b3c-4a15-8e0b-f775425bbf3c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "371a5796-9927-447e-9287-f0cd7428e855", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "20dcacf3c3cef6dd65a95411560c421efcfa6c501a727832b1e72c1df42f0ae5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b936443-cc79-4985-9670-5228ccde60b6", "node_type": "1", "metadata": {}, "hash": "da22b7ba743231e97503e1c29f8db4dea794aabfdcdcc7948f3d4cd4e7218d15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6582, "end_char_idx": 9041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's take 
another image to illustrate:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-prompted-im-captioning.jpg" alt="Image of the Eiffel Tower at night"/>
</div>

Photo by [Denys Nevozhai](https://unsplash.com/@dnevozhai).
   
Textual and image prompts can be passed to the model's processor as a single list to create appropriate inputs.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...     "This is an image of ",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
This is an image of the Eiffel Tower in Paris, France.
```

## Few-shot prompting

While IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with 
other restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.
By providing examples in the prompt, you can steer the model to generate results that mimic the format of given examples. 

Let's use the previous image of the Eiffel Tower as an example for the model and build a prompt that demonstrates to the model 
that in addition to learning what the object in an image is, we would also like to get some interesting information about it. 
Then, let's see, if we can get the same response format for an image of the Statue of Liberty:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg" alt="Image of the Statue of Liberty"/>
</div>

Photo by [Juan Mayobre](https://unsplash.com/@jmayobres).
  
```py
>>> prompt = ["User:",
...            "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...            "Describe this image.\nAssistant: An image of the Eiffel Tower at night.


key: _node_content
string_value:
{"id_": "8b936443-cc79-4985-9670-5228ccde60b6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "917f8815-8b3c-4a15-8e0b-f775425bbf3c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "144e08afd499cd92aa47ed8eb32ba072aaf36fc9ad3e458903c4429e9c6bc5b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aac9cd99-367c-45ff-a062-aebc1e1c55a3", "node_type": "1", "metadata": {}, "hash": "8ab4a099d58b7dd269830a0ff351a502f2e7bd7ef074339997e740a7193538d3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7713, "end_char_idx": 10618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

## Few-shot prompting

While IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with 
other restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.
By providing examples in the prompt, you can steer the model to generate results that mimic the format of given examples. 

Let's use the previous image of the Eiffel Tower as an example for the model and build a prompt that demonstrates to the model 
that in addition to learning what the object in an image is, we would also like to get some interesting information about it. 
Then, let's see, if we can get the same response format for an image of the Statue of Liberty:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg" alt="Image of the Statue of Liberty"/>
</div>

Photo by [Juan Mayobre](https://unsplash.com/@jmayobres).
  
```py
>>> prompt = ["User:",
...            "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...            "Describe this image.\nAssistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building.\n",
...            "User:",
...            "https://images.unsplash.com/photo-1524099163253-32b7f0256868?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3387&q=80",
...            "Describe this image.\nAssistant:"
...            ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=30, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
User: Describe this image.
Assistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. 
User: Describe this image.
Assistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.
```

Notice that just from a single example (i.e., 1-shot) the model has learned how to perform the task. For more complex tasks, 
feel free to experiment with a larger number of examples (e.g., 3-shot, 5-shot, etc.).

## Visual question answering

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image. Similar to image 
captioning it can be used in accessibility applications, but also in education (reasoning about visual materials), customer 
service (questions about products based on images), and image retrieval.


key: _node_content
string_value:
{"id_": "aac9cd99-367c-45ff-a062-aebc1e1c55a3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b936443-cc79-4985-9670-5228ccde60b6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f75502fb7b896da42126acd12ee81030221802a045555109fd0156cfa94eefe0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a52295e9-7bef-4cbb-839e-4b779c6dd7fb", "node_type": "1", "metadata": {}, "hash": "c8f66bfa86885f3915a179717b6d0338d4942e643b5475e337780cba910cd25b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9811, "end_char_idx": 12703, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Assistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. 
User: Describe this image.
Assistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.
```

Notice that just from a single example (i.e., 1-shot) the model has learned how to perform the task. For more complex tasks, 
feel free to experiment with a larger number of examples (e.g., 3-shot, 5-shot, etc.).

## Visual question answering

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image. Similar to image 
captioning it can be used in accessibility applications, but also in education (reasoning about visual materials), customer 
service (questions about products based on images), and image retrieval.

Let's get a new image for this task: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-vqa.jpg" alt="Image of a couple having a picnic"/>
</div>

Photo by [Jarritos Mexican Soda](https://unsplash.com/@jarritos). 

You can steer the model from image captioning to visual question answering by prompting it with appropriate instructions: 

```py
>>> prompt = [
...     "Instruction: Provide an answer to the question. Use the image to answer.\n",
...     "https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",
...     "Question: Where are these people and what's the weather like? Answer:"
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=20, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
Instruction: Provide an answer to the question. Use the image to answer.
 Question: Where are these people and what's the weather like? Answer: They're in a park in New York City, and it's a beautiful day.
```

## Image classification

IDEFICS is capable of classifying images into different categories without being explicitly trained on data containing 
labeled examples from those specific categories. Given a list of categories and using its image and text understanding 
capabilities, the model can infer which category the image likely belongs to. 

Say, we have this image of a vegetable stand: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-classification.jpg" alt="Image of a vegetable stand"/>
</div>

Photo by [Peter Wendt](https://unsplash.com/@peterwendt).


key: _node_content
string_value:
{"id_": "a52295e9-7bef-4cbb-839e-4b779c6dd7fb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aac9cd99-367c-45ff-a062-aebc1e1c55a3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ee4f0d3f7f8f3181ecda9e99d1a30d5b20c24cf926f75eafedaff3911916e286", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd1a5279-30aa-4b61-bdd3-8385fc20b22c", "node_type": "1", "metadata": {}, "hash": "20c5eabf06d276d9cda6a51d857c804a7c0d6e1057a6c11ff9896d4510ae0363", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11420, "end_char_idx": 14556, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Answer:"
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=20, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
Instruction: Provide an answer to the question. Use the image to answer.
 Question: Where are these people and what's the weather like? Answer: They're in a park in New York City, and it's a beautiful day.
```

## Image classification

IDEFICS is capable of classifying images into different categories without being explicitly trained on data containing 
labeled examples from those specific categories. Given a list of categories and using its image and text understanding 
capabilities, the model can infer which category the image likely belongs to. 

Say, we have this image of a vegetable stand: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-classification.jpg" alt="Image of a vegetable stand"/>
</div>

Photo by [Peter Wendt](https://unsplash.com/@peterwendt).

We can instruct the model to classify the image into one of the categories that we have:

```py
>>> categories = ['animals','vegetables', 'city landscape', 'cars', 'office']
>>> prompt = [f"Instruction: Classify the following image into a single category from the following list: {categories}.\n",
...     "https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",    
...     "Category: "
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=6, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
Instruction: Classify the following image into a single category from the following list: ['animals', 'vegetables', 'city landscape', 'cars', 'office'].
Category: Vegetables
```  

In the example above we instruct the model to classify the image into a single category, however, you can also prompt the model to do rank classification.

## Image-guided text generation

For more creative applications, you can use image-guided text generation to generate text based on an image. This can be 
useful to create descriptions of products, ads, descriptions of a scene, etc. 

Let's prompt IDEFICS to write a story based on a simple image of a red door: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-story-generation.jpg" alt="Image of a red door with a pumpkin on the steps"/>
</div>

Photo by [Craig Tidball](https://unsplash.com/@devonshiremedia).


key: _node_content
string_value:
{"id_": "dd1a5279-30aa-4b61-bdd3-8385fc20b22c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a52295e9-7bef-4cbb-839e-4b779c6dd7fb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "20c9189b3cdd657278e85f2490b51c98caefa22e292309c773d8e1c2cc4ddc90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c950c3d4-3410-47c9-86aa-0edeba4f1946", "node_type": "1", "metadata": {}, "hash": "76fcaeea0553411f5c081b1fc41752115607bd0eefda7001fc7ce7e42967a99c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13754, "end_char_idx": 16529, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Category: Vegetables
```  

In the example above we instruct the model to classify the image into a single category, however, you can also prompt the model to do rank classification.

## Image-guided text generation

For more creative applications, you can use image-guided text generation to generate text based on an image. This can be 
useful to create descriptions of products, ads, descriptions of a scene, etc. 

Let's prompt IDEFICS to write a story based on a simple image of a red door: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-story-generation.jpg" alt="Image of a red door with a pumpkin on the steps"/>
</div>

Photo by [Craig Tidball](https://unsplash.com/@devonshiremedia).
  
```py
>>> prompt = ["Instruction: Use the image to write a story. \n",
...     "https://images.unsplash.com/photo-1517086822157-2b0358e7684a?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2203&q=80",
...     "Story: \n"]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, num_beams=2, max_new_tokens=200, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0]) 
Instruction: Use the image to write a story. 
 Story: 
Once upon a time, there was a little girl who lived in a house with a red door.  She loved her red door.  It was the prettiest door in the whole world.

One day, the little girl was playing in her yard when she noticed a man standing on her doorstep.  He was wearing a long black coat and a top hat.

The little girl ran inside and told her mother about the man.

Her mother said, “Don’t worry, honey.  He’s just a friendly ghost.”

The little girl wasn’t sure if she believed her mother, but she went outside anyway.

When she got to the door, the man was gone.

The next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep.

He was wearing a long black coat and a top hat.

The little girl ran
```

Looks like IDEFICS noticed the pumpkin on the doorstep and went with a spooky Halloween story about a ghost.

<Tip>

For longer outputs like this, you will greatly benefit from tweaking the text generation strategy. This can help 
you significantly improve the quality of the generated output. Check out [Text generation strategies](../generation_strategies) 
to learn more. 
</Tip>

## Running inference in batch mode

All of the earlier sections illustrated IDEFICS for a single example.


key: _node_content
string_value:
{"id_": "c950c3d4-3410-47c9-86aa-0edeba4f1946", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd1a5279-30aa-4b61-bdd3-8385fc20b22c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "92dcc08b354148ecbb3aa741ee067da0e3c777478f46de22a062455318485ef4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "27fb390a-17a2-4625-9e5f-95aab180547a", "node_type": "1", "metadata": {}, "hash": "3caf6a68132596513471071bf1e0996a13ea2fc60dab38e30eb2d8189643694c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15282, "end_char_idx": 17893, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Story: 
Once upon a time, there was a little girl who lived in a house with a red door.  She loved her red door.  It was the prettiest door in the whole world.

One day, the little girl was playing in her yard when she noticed a man standing on her doorstep.  He was wearing a long black coat and a top hat.

The little girl ran inside and told her mother about the man.

Her mother said, “Don’t worry, honey.  He’s just a friendly ghost.”

The little girl wasn’t sure if she believed her mother, but she went outside anyway.

When she got to the door, the man was gone.

The next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep.

He was wearing a long black coat and a top hat.

The little girl ran
```

Looks like IDEFICS noticed the pumpkin on the doorstep and went with a spooky Halloween story about a ghost.

<Tip>

For longer outputs like this, you will greatly benefit from tweaking the text generation strategy. This can help 
you significantly improve the quality of the generated output. Check out [Text generation strategies](../generation_strategies) 
to learn more. 
</Tip>

## Running inference in batch mode

All of the earlier sections illustrated IDEFICS for a single example. In a very similar fashion, you can run inference 
for a batch of examples by passing a list of prompts:

```py
>>> prompts = [
...     [   "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...         "This is an image of ",
...     ],
...     [   "https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",
...         "This is an image of ",
...     ],
...     [   "https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",
...         "This is an image of ",
...     ],
... ]

>>> inputs = processor(prompts, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> for i,t in enumerate(generated_text):
...     print(f"{i}:\n{t}\n") 
0:
This is an image of the Eiffel Tower in Paris, France.

1:
This is an image of a couple on a picnic blanket.


key: _node_content
string_value:
{"id_": "27fb390a-17a2-4625-9e5f-95aab180547a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d7baa6-d241-4ac7-830a-a2f71b4ac012", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc0df803abbffbf6abd39cdf0f7a231135f5ac3095650bd2a7ebff0196457745", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c950c3d4-3410-47c9-86aa-0edeba4f1946", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics/", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1232072dafd42613cbe408e72dd402295758f92a602ad3de425ea1453a3c7cad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17841, "end_char_idx": 20184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1:
This is an image of a couple on a picnic blanket.

2:
This is an image of a vegetable stand.
```

## IDEFICS instruct for conversational use

For conversational use cases, you can find fine-tuned instructed versions of the model on the 🤗 Hub: 
`HuggingFaceM4/idefics-80b-instruct` and `HuggingFaceM4/idefics-9b-instruct`.

These checkpoints are the result of fine-tuning the respective base models on a mixture of supervised and instruction 
fine-tuning datasets, which boosts the downstream performance while making the models more usable in conversational settings.

The use and prompting for the conversational use is very similar to using the base models: 

```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor

>>> device = "cuda" if torch.cuda.is_available() else "cpu"

>>> checkpoint = "HuggingFaceM4/idefics-9b-instruct"
>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> prompts = [
...     [
...         "User: What is in this image?",
...         "https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG",
...         "<end_of_utterance>",

...         "\nAssistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground.<end_of_utterance>",

...         "\nUser:",
...         "https://static.wikia.nocookie.net/asterix/images/2/25/R22b.gif/revision/latest?cb=20110815073052",
...         "And who is that?<end_of_utterance>",

...         "\nAssistant:",
...     ],
... ]

>>> # --batched mode
>>> inputs = processor(prompts, add_end_of_utterance_token=False, return_tensors="pt").to(device)
>>> # --single sample mode
>>> # inputs = processor(prompts[0], return_tensors="pt").to(device)

>>> # Generation args
>>> exit_condition = processor.tokenizer("<end_of_utterance>", add_special_tokens=False).input_ids
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, eos_token_id=exit_condition, bad_words_ids=bad_words_ids, max_length=100)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> for i, t in enumerate(generated_text):
...     print(f"{i}:\n{t}\n")
```


key: _node_content
string_value:
{"id_": "47f4e7d4-a2d4-4084-8a31-3116cbdfd0fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "65024db8-cfe4-442c-9786-7fb838d43b10", "node_type": "1", "metadata": {}, "hash": "d69b9fbd8f133968faf60fad49c0cf076cb76dd8d11684fbd15c54d237a1639c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Summarization

[[open-in-colab]]

<Youtube id="yHnr5Dk2zCI"/>

Summarization creates a shorter version of a document or an article that captures all the important information. Along with translation, it is another example of a task that can be formulated as a sequence-to-sequence task. Summarization can be:

- Extractive: extract the most relevant information from a document.
- Abstractive: generate new text that captures the most relevant information.

This guide will show you how to:

1. Finetune [T5](https://huggingface.co/google-t5/t5-small) on the California state bill subset of the [BillSum](https://huggingface.co/datasets/billsum) dataset for abstractive summarization.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/summarization)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate rouge_score
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load BillSum dataset

Start by loading the smaller California state bill subset of the BillSum dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> billsum = load_dataset("billsum", split="ca_test")
```

Split the dataset into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> billsum = billsum.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> billsum["train"][0]
{'summary': 'Existing law authorizes state agencies to enter into contracts for the acquisition of goods or services upon approval by the Department of General Services. Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\nThis bill would provide that no reimbursement is required by this act for a specified reason.',
 'text': 'The people of the State of California do enact as follows:\n\n\nSECTION 1.\nSection 10295.35 is added to the Public Contract Code, to read:\n10295.35.\n(a) (1) Notwithstanding any other law,


key: _node_content
string_value:
{"id_": "65024db8-cfe4-442c-9786-7fb838d43b10", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47f4e7d4-a2d4-4084-8a31-3116cbdfd0fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b7ec9ec3402a7b5d34997287dadba97bad8a49297d5c1924710b29a741fcfdb9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa22029b-c54c-4e14-af39-e60394f97e5a", "node_type": "1", "metadata": {}, "hash": "c9ac8e7815f0574e35d09a6dd26f2828a1c4909300a12220195a21dade7bd7da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1919, "end_char_idx": 5868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\nThis bill would provide that no reimbursement is required by this act for a specified reason.',
 'text': 'The people of the State of California do enact as follows:\n\n\nSECTION 1.\nSection 10295.35 is added to the Public Contract Code, to read:\n10295.35.\n(a) (1) Notwithstanding any other law, a state agency shall not enter into any contract for the acquisition of goods or services in the amount of one hundred thousand dollars ($100,000) or more with a contractor that, in the provision of benefits, discriminates between employees on the basis of an employee’s or dependent’s actual or perceived gender identity, including, but not limited to, the employee’s or dependent’s identification as transgender.\n(2) For purposes of this section, “contract” includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\n(4) The requirements of this section shall apply only to those portions of a contractor’s operations that occur under any of the following conditions:\n(A) Within the state.\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor’s presence at that location is connected to a contract with the state.\n(C) Elsewhere in the United States where work related to a state contract is being performed.\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor’s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with,


key: _node_content
string_value:
{"id_": "aa22029b-c54c-4e14-af39-e60394f97e5a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65024db8-cfe4-442c-9786-7fb838d43b10", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "95b7580165f235f0d72e12b6c8f5300ebcc896670950795cdfa4abea907133a3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e15ca35-0d9f-4e30-a7a8-97b73ad6d2a5", "node_type": "1", "metadata": {}, "hash": "d06e8648aeacf19c052162a6d4ea3dd77baecd5b96e99a2427822bf7e0b9c80b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3855, "end_char_idx": 7575, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
but not limited to, the employee’s or dependent’s identification as transgender.\n(2) For purposes of this section, “contract” includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\n(4) The requirements of this section shall apply only to those portions of a contractor’s operations that occur under any of the following conditions:\n(A) Within the state.\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor’s presence at that location is connected to a contract with the state.\n(C) Elsewhere in the United States where work related to a state contract is being performed.\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor’s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.


key: _node_content
string_value:
{"id_": "3e15ca35-0d9f-4e30-a7a8-97b73ad6d2a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa22029b-c54c-4e14-af39-e60394f97e5a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "615fe67aa4ac700c7783f00f0ec0bbbf7fab8d96cb5eb17b47fa87a17a409965", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83a867c0-d769-4153-81b6-60d39cd7f4c6", "node_type": "1", "metadata": {}, "hash": "e184f27f91ed33251037adbf26dbdde4ba8d1122c0f70d760ea4c9dc986bdfb1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5541, "end_char_idx": 9360, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\nSEC. 2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}


key: _node_content
string_value:
{"id_": "83a867c0-d769-4153-81b6-60d39cd7f4c6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e15ca35-0d9f-4e30-a7a8-97b73ad6d2a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a953bcb09cf5cf956aa887441955dfdc8a3dd58bc4a7121864c489dca6c38622", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c98c151-e4c2-40e4-be95-fcd549e3d41a", "node_type": "1", "metadata": {}, "hash": "bdd516d048cb095cdf53b9aa952df7a6cbe211995fb48bd543965af445200623", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7442, "end_char_idx": 10203, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\nSEC. 2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}
```

There are two fields that you'll want to use:

- `text`: the text of the bill which'll be the input to the model.
- `summary`: a condensed version of `text` which'll be the model target.

## Preprocess

The next step is to load a T5 tokenizer to process `text` and `summary`:

```py
>>> from transformers import AutoTokenizer

>>> checkpoint = "google-t5/t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
```

The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a summarization task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Use the keyword `text_target` argument when tokenizing labels.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
>>> prefix = "summarize: "


key: _node_content
string_value:
{"id_": "0c98c151-e4c2-40e4-be95-fcd549e3d41a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83a867c0-d769-4153-81b6-60d39cd7f4c6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "26118ede7494acdcb0fd6b85960d974297318463a45e550b5b4fcf81290f0206", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3173d32c-2e49-42ef-97c7-c20cc144414c", "node_type": "1", "metadata": {}, "hash": "c67ea74fee14e2d3caffe00d7a6ea3311b1f5f8c114cbef7d0fef2f8de4a0fe6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8510, "end_char_idx": 12162, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}
```

There are two fields that you'll want to use:

- `text`: the text of the bill which'll be the input to the model.
- `summary`: a condensed version of `text` which'll be the model target.

## Preprocess

The next step is to load a T5 tokenizer to process `text` and `summary`:

```py
>>> from transformers import AutoTokenizer

>>> checkpoint = "google-t5/t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
```

The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a summarization task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Use the keyword `text_target` argument when tokenizing labels.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
>>> prefix = "summarize: "


>>> def preprocess_function(examples):
...     inputs = [prefix + doc for doc in examples["text"]]
...     model_inputs = tokenizer(inputs, max_length=1024, truncation=True)

...     labels = tokenizer(text_target=examples["summary"], max_length=128, truncation=True)

...     model_inputs["labels"] = labels["input_ids"]
...     return model_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_billsum = billsum.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorForSeq2Seq`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>

```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)
```
</pt>
<tf>

```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [ROUGE](https://huggingface.co/spaces/evaluate-metric/rouge) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> rouge = evaluate.load("rouge")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the ROUGE metric:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "3173d32c-2e49-42ef-97c7-c20cc144414c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c98c151-e4c2-40e4-be95-fcd549e3d41a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "eb8b936a0177abeb30a8feff14ae4d0cb697f9c2a3ac34a1cfc5a6238ab700a3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f14a6248-deb5-4dc2-a3fb-058b91f50b7b", "node_type": "1", "metadata": {}, "hash": "1fc095916afa675f55c71c898912f5e8a12a05820a51c9f84f0ce208ff8be328", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12165, "end_char_idx": 15070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

...     result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)

...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]
...     result["gen_len"] = np.mean(prediction_lens)

...     return {k: round(v, 4) for k, v in result.items()}
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:

```py
>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the ROUGE metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_billsum_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=4,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_billsum["train"],
...     eval_dataset=tokenized_billsum["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "f14a6248-deb5-4dc2-a3fb-058b91f50b7b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3173d32c-2e49-42ef-97c7-c20cc144414c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5bc3903857b88f9fdc059562a21b14565af69084b098f3ad800a2830d9d89621", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "89b7cf3b-379d-40a3-9d67-f6f71e6b9736", "node_type": "1", "metadata": {}, "hash": "76f7488be7188048a3312e3b809ec8bb3c8e382229592e80113100cfe3b302de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13570, "end_char_idx": 16590, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the ROUGE metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_billsum_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=4,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_billsum["train"],
...     eval_dataset=tokenized_billsum["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_billsum["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_billsum["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the ROUGE score from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).


key: _node_content
string_value:
{"id_": "89b7cf3b-379d-40a3-9d67-f6f71e6b9736", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f14a6248-deb5-4dc2-a3fb-058b91f50b7b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7bacd3cae759d37af180eae4e3ace2952de6e71b8145e31651922c4eeddf6af4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a926875c-debb-4b46-af9f-0574b9ca5f56", "node_type": "1", "metadata": {}, "hash": "063f3f076384dff4a8b28e20d5c36e2d8758bfd57f5946327c7d3786d17d860d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15072, "end_char_idx": 18303, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_billsum["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_billsum["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the ROUGE score from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_billsum_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for summarization, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on.


key: _node_content
string_value:
{"id_": "a926875c-debb-4b46-af9f-0574b9ca5f56", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89b7cf3b-379d-40a3-9d67-f6f71e6b9736", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "378f48d97d6667f217c94b4407d20152146cbba4b65f2751d067b0cd4d2c1eea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87c3bb1b-fe15-4549-a140-f82cf6aef859", "node_type": "1", "metadata": {}, "hash": "16dcb01e3c780f50e688b261f9a493f76dbd2d759ed0de3a5afbbca6c2c676fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17302, "end_char_idx": 19550, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for summarization, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on. For summarization you should prefix your input as shown below:

```py
>>> text = "summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for summarization with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> summarizer = pipeline("summarization", model="stevhliu/my_awesome_billsum_model")
>>> summarizer(text)
[{"summary_text": "The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country."}]
```

You can also manually replicate the results of the `pipeline` if you'd like:


key: _node_content
string_value:
{"id_": "87c3bb1b-fe15-4549-a140-f82cf6aef859", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35d6de67-5e75-479c-98c1-0a672c1e80e5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f7e777b30e2d693ad13cea89b1140c255394f7fd5ba6a6941db177f72249b63", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a926875c-debb-4b46-af9f-0574b9ca5f56", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization/", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4b17273276821bdd673c1b87c70a3293a691529659887f2a5ad147b37a04dab3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18082, "end_char_idx": 21890, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on. For summarization you should prefix your input as shown below:

```py
>>> text = "summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for summarization with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> summarizer = pipeline("summarization", model="stevhliu/my_awesome_billsum_model")
>>> summarizer(text)
[{"summary_text": "The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country."}]
```

You can also manually replicate the results of the `pipeline` if you'd like:


<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> inputs = tokenizer(text, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import AutoModelForSeq2SeqLM

>>> model = AutoModelForSeq2SeqLM.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=False)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'the inflation reduction act lowers prescription drug costs, health care costs, and energy costs. it's the most aggressive action on tackling the climate crisis in american history. it will ask the ultra-wealthy and corporations to pay their fair share.'
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> inputs = tokenizer(text, return_tensors="tf").input_ids
```

Use the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=False)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'the inflation reduction act lowers prescription drug costs, health care costs, and energy costs. it's the most aggressive action on tackling the climate crisis in american history. it will ask the ultra-wealthy and corporations to pay their fair share.'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8693000a-f91b-43e3-b991-cb3bf3b4da7f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f00d0d32-8d5b-4907-99b9-6188dc2a92b6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8b18248b5a377a3dc2fd14d64747d7a942be9dd863b5fa888be602c1603b157", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ceae888-7e09-4be3-aa23-b01582cb7c4c", "node_type": "1", "metadata": {}, "hash": "6ec150254f33a00b4c0197617151c016cd3de138c6dfe16f684383b42f7ecf07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Mask Generation

Mask generation is the task of generating semantically meaningful masks for an image. 
This task is very similar to [image segmentation](semantic_segmentation), but many differences exist. Image segmentation models are trained on labeled datasets and are limited to the classes they have seen during training; they return a set of masks and corresponding classes, given an image. 

Mask generation models are trained on large amounts of data and operate in two modes. 
- Prompting mode: In this mode, the model takes in an image and a prompt, where a prompt can be a 2D point location (XY coordinates) in the image within an object or a bounding box surrounding an object. In prompting mode, the model only returns the mask over the object 
that the prompt is pointing out. 
- Segment Everything mode: In segment everything, given an image, the model generates every mask in the image. To do so, a grid of points is generated and overlaid on the image for inference. 

Mask generation task is supported by [Segment Anything Model (SAM)](model_doc/sam). It's a powerful model that consists of a Vision Transformer-based image encoder, a prompt encoder, and a two-way transformer mask decoder. Images and prompts are encoded, and the decoder takes these embeddings and generates valid masks. 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sam.png" alt="SAM Architecture"/>
</div>

SAM serves as a powerful foundation model for segmentation as it has large data coverage. It is trained on 
[SA-1B](https://ai.meta.com/datasets/segment-anything/), a dataset with 1 million images and 1.1 billion masks. 

In this guide, you will learn how to:
- Infer in segment everything mode with batching,
- Infer in point prompting mode,
- Infer in box prompting mode.

First, let's install `transformers`:

```bash
pip install -q transformers
```

## Mask Generation Pipeline

The easiest way to infer mask generation models is to use the `mask-generation` pipeline.

```python
>>> from transformers import pipeline

>>> checkpoint = "facebook/sam-vit-base"
>>> mask_generator = pipeline(model=checkpoint, task="mask-generation")
```

Let's see the image.

```python
from PIL import Image
import requests

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Example Image"/>
</div>

Let's segment everything. `points-per-batch` enables parallel inference of points in segment everything mode. This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.


key: _node_content
string_value:
{"id_": "2ceae888-7e09-4be3-aa23-b01582cb7c4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f00d0d32-8d5b-4907-99b9-6188dc2a92b6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8b18248b5a377a3dc2fd14d64747d7a942be9dd863b5fa888be602c1603b157", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8693000a-f91b-43e3-b991-cb3bf3b4da7f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5ad3a3cd195c3f26d9d799dbde120b68dc2bbbed72904ab4913553874a095352", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5cf394be-2327-4c05-8410-5115b22768ba", "node_type": "1", "metadata": {}, "hash": "c099248979b7f82b2d3fc917922ca409f250c854bb01029223269d4b40c174c5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1311, "end_char_idx": 4262, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sam.png" alt="SAM Architecture"/>
</div>

SAM serves as a powerful foundation model for segmentation as it has large data coverage. It is trained on 
[SA-1B](https://ai.meta.com/datasets/segment-anything/), a dataset with 1 million images and 1.1 billion masks. 

In this guide, you will learn how to:
- Infer in segment everything mode with batching,
- Infer in point prompting mode,
- Infer in box prompting mode.

First, let's install `transformers`:

```bash
pip install -q transformers
```

## Mask Generation Pipeline

The easiest way to infer mask generation models is to use the `mask-generation` pipeline.

```python
>>> from transformers import pipeline

>>> checkpoint = "facebook/sam-vit-base"
>>> mask_generator = pipeline(model=checkpoint, task="mask-generation")
```

Let's see the image.

```python
from PIL import Image
import requests

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Example Image"/>
</div>

Let's segment everything. `points-per-batch` enables parallel inference of points in segment everything mode. This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.

```python
masks = mask_generator(image, points_per_batch=128, pred_iou_thresh=0.88)
```

The `masks` looks like the following:

```bash
{'masks': [array([[False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         ...,
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False]]),
  array([[False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         ...,
'scores': tensor([0.9972, 0.9917,
        ...,
}
```

We can visualize them like this:

```python
import matplotlib.pyplot as plt

plt.imshow(image, cmap='gray')

for i, mask in enumerate(masks["masks"]):
    plt.imshow(mask, cmap='viridis', alpha=0.1, vmin=0, vmax=1)

plt.axis('off')
plt.show()
```

Below is the original image in grayscale with colorful maps overlaid. Very impressive.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_segmented.png" alt="Visualized"/>
</div>


key: _node_content
string_value:
{"id_": "5cf394be-2327-4c05-8410-5115b22768ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f00d0d32-8d5b-4907-99b9-6188dc2a92b6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8b18248b5a377a3dc2fd14d64747d7a942be9dd863b5fa888be602c1603b157", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ceae888-7e09-4be3-aa23-b01582cb7c4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c102d308e682e3bc7f159ac853e77c567ed58a75be69abceb19fdb52bf42dd7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6ce3989-5f1f-4239-b8e6-e14edb45dda8", "node_type": "1", "metadata": {}, "hash": "f0ea727a0aa4d79b68a673172a4fea6e31fea884671222f24da93b79c0fc4de1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2764, "end_char_idx": 6701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.

```python
masks = mask_generator(image, points_per_batch=128, pred_iou_thresh=0.88)
```

The `masks` looks like the following:

```bash
{'masks': [array([[False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         ...,
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False]]),
  array([[False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         ...,
'scores': tensor([0.9972, 0.9917,
        ...,
}
```

We can visualize them like this:

```python
import matplotlib.pyplot as plt

plt.imshow(image, cmap='gray')

for i, mask in enumerate(masks["masks"]):
    plt.imshow(mask, cmap='viridis', alpha=0.1, vmin=0, vmax=1)

plt.axis('off')
plt.show()
```

Below is the original image in grayscale with colorful maps overlaid. Very impressive.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_segmented.png" alt="Visualized"/>
</div>


## Model Inference

### Point Prompting

You can also use the model without the pipeline. To do so, initialize the model and
the processor.

```python
from transformers import SamModel, SamProcessor
import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = SamModel.from_pretrained("facebook/sam-vit-base").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-base")
```

To do point prompting, pass the input point to the processor, then take the processor output
and pass it to the model for inference. To post-process the model output, pass the outputs and
`original_sizes` and `reshaped_input_sizes` we take from the processor's initial output. We need to pass these 
since the processor resizes the image, and the output needs to be extrapolated.

```python
input_points = [[[2592, 1728]]] # point location of the bee

inputs = processor(image, input_points=input_points, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)
masks = processor.image_processor.post_process_masks(outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu())
```
We can visualize the three masks in the `masks` output.

```python
import matplotlib.pyplot as plt
import numpy as np

fig, axes = plt.subplots(1, 4, figsize=(15, 5))

axes[0].imshow(image)
axes[0].set_title('Original Image')
mask_list = [masks[0][0][0].numpy(), masks[0][0][1].numpy(), masks[0][0][2].numpy()]

for i, mask in enumerate(mask_list, start=1):
    overlayed_image = np.array(image).copy()

    overlayed_image[:,:,0] = np.where(mask == 1, 255, overlayed_image[:,:,0])
    overlayed_image[:,:,1] = np.where(mask == 1, 0, overlayed_image[:,:,1])
    overlayed_image[:,:,2] = np.where(mask == 1, 0, overlayed_image[:,:,2])
    
    axes[i].imshow(overlayed_image)
    axes[i].set_title(f'Mask {i}')
for ax in axes:
    ax.axis('off')

plt.show()
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/masks.png" alt="Visualized"/>
</div>

### Box Prompting

You can also do box prompting in a similar fashion to point prompting. You can simply pass the input box in the format of a list
`[x_min, y_min, x_max, y_max]` format along with the image to the `processor`. Take the processor output and directly pass it 
to the model, then post-process the output again.


key: _node_content
string_value:
{"id_": "b6ce3989-5f1f-4239-b8e6-e14edb45dda8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f00d0d32-8d5b-4907-99b9-6188dc2a92b6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8b18248b5a377a3dc2fd14d64747d7a942be9dd863b5fa888be602c1603b157", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5cf394be-2327-4c05-8410-5115b22768ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation/", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "615ac588742916c91951a7ee3da0dc01b50e54c0b6fe9f8e1cd703ae5ad4bbf5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6704, "end_char_idx": 7995, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# bounding box around the bee
box = [2350, 1600, 2850, 2100]

inputs = processor(
        image,
        input_boxes=[[[box]]],
        return_tensors="pt"
    ).to("cuda")

with torch.no_grad():
    outputs = model(**inputs)

mask = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu()
)[0][0][0].numpy()
```

You can visualize the bounding box around the bee as shown below.

```python
import matplotlib.patches as patches

fig, ax = plt.subplots()
ax.imshow(image)

rectangle = patches.Rectangle((2350, 1600), 500, 500, linewidth=2, edgecolor='r', facecolor='none')
ax.add_patch(rectangle)
ax.axis("off")
plt.show()
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/bbox.png" alt="Visualized Bbox"/>
</div>

You can see the inference output below. 

```python
fig, ax = plt.subplots()
ax.imshow(image)
ax.imshow(mask, cmap='viridis', alpha=0.4)

ax.axis("off")
plt.show()
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/box_inference.png" alt="Visualized Inference"/>
</div>


key: _node_content
string_value:
{"id_": "7cd8c38a-46f8-4110-8696-833b7d4f83be", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3db0e33f-362a-4ab0-9bd6-be94cf6bbb26", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b8acc91b27c14643b50d840ad607fcdea52eb545598ffff0fde91029f3e31f1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15ae85f0-5953-46f6-8fec-e6768fe5fe3d", "node_type": "1", "metadata": {}, "hash": "57f0f676a9f4b24e604e3b97d0e1f70e4861a8cb761b5e925fbca7da301c5826", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2898, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Zero-shot object detection

[[open-in-colab]]

Traditionally, models used for [object detection](object_detection) require labeled image datasets for training,
and are limited to detecting the set of classes from the training data.

Zero-shot object detection is supported by the [OWL-ViT](../model_doc/owlvit) model which uses a different approach. OWL-ViT
is an open-vocabulary object detector. It means that it can detect objects in images based on free-text queries without
the need to fine-tune the model on labeled datasets.

OWL-ViT leverages multi-modal representations to perform open-vocabulary detection. It combines [CLIP](../model_doc/clip) with
lightweight object classification and localization heads. Open-vocabulary detection is achieved by embedding free-text queries with the text encoder of CLIP and using them as input to the object classification and localization heads.
associate images and their corresponding textual descriptions, and ViT processes image patches as inputs. The authors
of OWL-ViT first trained CLIP from scratch and then fine-tuned OWL-ViT end to end on standard object detection datasets using
a bipartite matching loss.

With this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.

In this guide, you will learn how to use OWL-ViT:
- to detect objects based on text prompts
- for batch object detection
- for image-guided object detection

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q transformers
```

## Zero-shot object detection pipeline

The simplest way to try out inference with OWL-ViT is to use it in a [`pipeline`]. Instantiate a pipeline
for zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):

```python
>>> from transformers import pipeline

>>> checkpoint = "google/owlv2-base-patch16-ensemble"
>>> detector = pipeline(model=checkpoint, task="zero-shot-object-detection")
```

Next, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is
a part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset.

```py
>>> import skimage
>>> import numpy as np
>>> from PIL import Image

>>> image = skimage.data.astronaut()
>>> image = Image.fromarray(np.uint8(image)).convert("RGB")

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png" alt="Astronaut Eileen Collins"/>
</div>

Pass the image and the candidate object labels to look for to the pipeline.
Here we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for.


key: _node_content
string_value:
{"id_": "15ae85f0-5953-46f6-8fec-e6768fe5fe3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3db0e33f-362a-4ab0-9bd6-be94cf6bbb26", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b8acc91b27c14643b50d840ad607fcdea52eb545598ffff0fde91029f3e31f1f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7cd8c38a-46f8-4110-8696-833b7d4f83be", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "469210e1a58964af0d1fa1ca50545cfa7835b79bca7240a229936bb81bcd8b50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc3b4a97-86b0-4448-80b8-dcf763d1c592", "node_type": "1", "metadata": {}, "hash": "0bedb9152c57476bafe862cc8c56f580398b1ab2552580308d8d30a181821169", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1167, "end_char_idx": 4636, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.

In this guide, you will learn how to use OWL-ViT:
- to detect objects based on text prompts
- for batch object detection
- for image-guided object detection

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q transformers
```

## Zero-shot object detection pipeline

The simplest way to try out inference with OWL-ViT is to use it in a [`pipeline`]. Instantiate a pipeline
for zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):

```python
>>> from transformers import pipeline

>>> checkpoint = "google/owlv2-base-patch16-ensemble"
>>> detector = pipeline(model=checkpoint, task="zero-shot-object-detection")
```

Next, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is
a part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset.

```py
>>> import skimage
>>> import numpy as np
>>> from PIL import Image

>>> image = skimage.data.astronaut()
>>> image = Image.fromarray(np.uint8(image)).convert("RGB")

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png" alt="Astronaut Eileen Collins"/>
</div>

Pass the image and the candidate object labels to look for to the pipeline.
Here we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for. 

```py
>>> predictions = detector(
...     image,
...     candidate_labels=["human face", "rocket", "nasa badge", "star-spangled banner"],
... )
>>> predictions
[{'score': 0.3571370542049408,
  'label': 'human face',
  'box': {'xmin': 180, 'ymin': 71, 'xmax': 271, 'ymax': 178}},
 {'score': 0.28099656105041504,
  'label': 'nasa badge',
  'box': {'xmin': 129, 'ymin': 348, 'xmax': 206, 'ymax': 427}},
 {'score': 0.2110239565372467,
  'label': 'rocket',
  'box': {'xmin': 350, 'ymin': -1, 'xmax': 468, 'ymax': 288}},
 {'score': 0.13790413737297058,
  'label': 'star-spangled banner',
  'box': {'xmin': 1, 'ymin': 1, 'xmax': 105, 'ymax': 509}},
 {'score': 0.11950037628412247,
  'label': 'nasa badge',
  'box': {'xmin': 277, 'ymin': 338, 'xmax': 327, 'ymax': 380}},
 {'score': 0.10649408400058746,
  'label': 'rocket',
  'box': {'xmin': 358, 'ymin': 64, 'xmax': 424, 'ymax': 280}}]
```

Let's visualize the predictions:

```py
>>> from PIL import ImageDraw

>>> draw = ImageDraw.Draw(image)

>>> for prediction in predictions:
...     box = prediction["box"]
...     label = prediction["label"]
...     score = prediction["score"]

...     xmin, ymin, xmax, ymax = box.values()
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{label}: {round(score,2)}", fill="white")

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_2.png" alt="Visualized predictions on NASA image"/>
</div>

## Text-prompted zero-shot object detection by hand

Now that you've seen how to use the zero-shot object detection pipeline, let's replicate the same
result manually.


key: _node_content
string_value:
{"id_": "dc3b4a97-86b0-4448-80b8-dcf763d1c592", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3db0e33f-362a-4ab0-9bd6-be94cf6bbb26", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b8acc91b27c14643b50d840ad607fcdea52eb545598ffff0fde91029f3e31f1f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15ae85f0-5953-46f6-8fec-e6768fe5fe3d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b19cc1e36d015714a2a4a0e0757fc176f8e9f55077cbcda5c2a47b90f16b0e3c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "98f07169-0274-4525-979f-d6a3860fffd5", "node_type": "1", "metadata": {}, "hash": "f2679b3e1283b8389acede742e3aa00aca7adae05978ae60bf292065ec2d21f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4638, "end_char_idx": 7506, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint)
>>> processor = AutoProcessor.from_pretrained(checkpoint)
```

Let's take a different image to switch things up.

```py
>>> import requests

>>> url = "https://unsplash.com/photos/oj0zeY2Ltk4/download?ixid=MnwxMjA3fDB8MXxzZWFyY2h8MTR8fHBpY25pY3xlbnwwfHx8fDE2Nzc0OTE1NDk&force=true&w=640"
>>> im = Image.open(requests.get(url, stream=True).raw)
>>> im
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_3.png" alt="Beach photo"/>
</div>

Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the
image for the model by resizing and normalizing it, and a [`CLIPTokenizer`] that takes care of the text inputs.

```py
>>> text_queries = ["hat", "book", "sunglasses", "camera"]
>>> inputs = processor(text=text_queries, images=im, return_tensors="pt")
```

Pass the inputs through the model, post-process, and visualize the results. Since the image processor resized images before
feeding them to the model, you need to use the [`~OwlViTImageProcessor.post_process_object_detection`] method to make sure the predicted bounding
boxes have the correct coordinates relative to the original image:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = torch.tensor([im.size[::-1]])
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(im)

>>> scores = results["scores"].tolist()
>>> labels = results["labels"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[label]}: {round(score,2)}", fill="white")

>>> im
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Batch processing

You can pass multiple sets of images and text queries to search for different (or same) objects in several images.
Let's use both an astronaut image and the beach image together.
For batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,
PyTorch tensors, or NumPy arrays.


key: _node_content
string_value:
{"id_": "98f07169-0274-4525-979f-d6a3860fffd5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3db0e33f-362a-4ab0-9bd6-be94cf6bbb26", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b8acc91b27c14643b50d840ad607fcdea52eb545598ffff0fde91029f3e31f1f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc3b4a97-86b0-4448-80b8-dcf763d1c592", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a666579b85349e282b6ff5a61fd596147b1a5474fd3ded7aabafed592b6086af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5aa8eab1-d94b-4424-af05-0def42caf355", "node_type": "1", "metadata": {}, "hash": "7ec40982ef3dc2111c501cdfbc6684f788849a2fc270c6be455be431fb0c863c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5996, "end_char_idx": 9215, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since the image processor resized images before
feeding them to the model, you need to use the [`~OwlViTImageProcessor.post_process_object_detection`] method to make sure the predicted bounding
boxes have the correct coordinates relative to the original image:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = torch.tensor([im.size[::-1]])
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(im)

>>> scores = results["scores"].tolist()
>>> labels = results["labels"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[label]}: {round(score,2)}", fill="white")

>>> im
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Batch processing

You can pass multiple sets of images and text queries to search for different (or same) objects in several images.
Let's use both an astronaut image and the beach image together.
For batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,
PyTorch tensors, or NumPy arrays.

```py
>>> images = [image, im]
>>> text_queries = [
...     ["human face", "rocket", "nasa badge", "star-spangled banner"],
...     ["hat", "book", "sunglasses", "camera"],
... ]
>>> inputs = processor(text=text_queries, images=images, return_tensors="pt")
```

Previously for post-processing you passed the single image's size as a tensor, but you can also pass a tuple, or, in case
of several images, a list of tuples. Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`).

```py
>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = [x.size[::-1] for x in images]
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)

>>> image_idx = 1
>>> draw = ImageDraw.Draw(images[image_idx])

>>> scores = results[image_idx]["scores"].tolist()
>>> labels = results[image_idx]["labels"].tolist()
>>> boxes = results[image_idx]["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[image_idx][label]}: {round(score,2)}", fill="white")

>>> images[image_idx]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Image-guided object detection

In addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means
you can use an image query to find similar objects in the target image.


key: _node_content
string_value:
{"id_": "5aa8eab1-d94b-4424-af05-0def42caf355", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3db0e33f-362a-4ab0-9bd6-be94cf6bbb26", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b8acc91b27c14643b50d840ad607fcdea52eb545598ffff0fde91029f3e31f1f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98f07169-0274-4525-979f-d6a3860fffd5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "27291f44e8dcec97b0241d351db8d6df4a3db2d97858d10c1ef2c7e35c185e1b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b42b45d-26b2-43dc-a3dd-3fbc9921fecb", "node_type": "1", "metadata": {}, "hash": "579fbd9f59083f4ea1aafad225d23c489c474f135b633e9077cf401783306302", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7929, "end_char_idx": 10432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`).

```py
>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = [x.size[::-1] for x in images]
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)

>>> image_idx = 1
>>> draw = ImageDraw.Draw(images[image_idx])

>>> scores = results[image_idx]["scores"].tolist()
>>> labels = results[image_idx]["labels"].tolist()
>>> boxes = results[image_idx]["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[image_idx][label]}: {round(score,2)}", fill="white")

>>> images[image_idx]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Image-guided object detection

In addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means
you can use an image query to find similar objects in the target image.
Unlike text queries, only a single example image is allowed.

Let's take an image with two cats on a couch as a target image, and an image of a single cat
as a query:

```py
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image_target = Image.open(requests.get(url, stream=True).raw)

>>> query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
>>> query_image = Image.open(requests.get(query_url, stream=True).raw)
```

Let's take a quick look at the images:

```py
>>> import matplotlib.pyplot as plt

>>> fig, ax = plt.subplots(1, 2)
>>> ax[0].imshow(image_target)
>>> ax[1].imshow(query_image)
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png" alt="Cats"/>
</div>

In the preprocessing step, instead of text queries, you now need to use `query_images`:

```py
>>> inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")
```

For predictions, instead of passing the inputs to the model, pass them to [`~OwlViTForObjectDetection.image_guided_detection`]. Draw the predictions
as before except now there are no labels.


key: _node_content
string_value:
{"id_": "2b42b45d-26b2-43dc-a3dd-3fbc9921fecb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3db0e33f-362a-4ab0-9bd6-be94cf6bbb26", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b8acc91b27c14643b50d840ad607fcdea52eb545598ffff0fde91029f3e31f1f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5aa8eab1-d94b-4424-af05-0def42caf355", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection/", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "75026e113e8fa4b5ee3693a0c420b72589fc75da24bac8b435fdb6d9fd70fdaa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9133, "end_char_idx": 11224, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means
you can use an image query to find similar objects in the target image.
Unlike text queries, only a single example image is allowed.

Let's take an image with two cats on a couch as a target image, and an image of a single cat
as a query:

```py
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image_target = Image.open(requests.get(url, stream=True).raw)

>>> query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
>>> query_image = Image.open(requests.get(query_url, stream=True).raw)
```

Let's take a quick look at the images:

```py
>>> import matplotlib.pyplot as plt

>>> fig, ax = plt.subplots(1, 2)
>>> ax[0].imshow(image_target)
>>> ax[1].imshow(query_image)
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png" alt="Cats"/>
</div>

In the preprocessing step, instead of text queries, you now need to use `query_images`:

```py
>>> inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")
```

For predictions, instead of passing the inputs to the model, pass them to [`~OwlViTForObjectDetection.image_guided_detection`]. Draw the predictions
as before except now there are no labels.

```py
>>> with torch.no_grad():
...     outputs = model.image_guided_detection(**inputs)
...     target_sizes = torch.tensor([image_target.size[::-1]])
...     results = processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(image_target)

>>> scores = results["scores"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="white", width=4)

>>> image_target
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_6.png" alt="Cats with bounding boxes"/>
</div>


key: _node_content
string_value:
{"id_": "4b2c1b62-a719-48fc-9be7-dd23ddae08af", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "356e94a0-c248-4a1b-a2c8-966c554443f4", "node_type": "1", "metadata": {}, "hash": "6124ef1c5d9dbf649af3c2f030fd53cd1b84bcb0c53bccb420bccefdfb79a230", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2327, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image Segmentation

[[open-in-colab]]

<Youtube id="dKE8SIt9C-w"/>

Image segmentation models separate areas corresponding to different areas of interest in an image. These models work by assigning a label to each pixel. There are several types of segmentation: semantic segmentation, instance segmentation, and panoptic segmentation.

In this guide, we will:
1. [Take a look at different types of segmentation](#types-of-segmentation).
2. [Have an end-to-end fine-tuning example for semantic segmentation](#fine-tuning-a-model-for-segmentation).

Before you begin, make sure you have all the necessary libraries installed:

```py
# uncomment to install the necessary libraries
!pip install -q datasets transformers evaluate accelerate
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Types of Segmentation

Semantic segmentation assigns a label or class to every single pixel in an image. Let's take a look at a semantic segmentation model output. It will assign the same class to every instance of an object it comes across in an image, for example, all cats will be labeled as "cat" instead of "cat-1", "cat-2".
We can use transformers' image segmentation pipeline to quickly infer a semantic segmentation model. Let's take a look at the example image.

```python
from transformers import pipeline
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg" alt="Segmentation Input"/>
</div>

We will use [nvidia/segformer-b1-finetuned-cityscapes-1024-1024](https://huggingface.co/nvidia/segformer-b1-finetuned-cityscapes-1024-1024).

```python
semantic_segmentation = pipeline("image-segmentation", "nvidia/segformer-b1-finetuned-cityscapes-1024-1024")
results = semantic_segmentation(image)
results
```

The segmentation pipeline output includes a mask for every predicted class.


key: _node_content
string_value:
{"id_": "356e94a0-c248-4a1b-a2c8-966c554443f4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b2c1b62-a719-48fc-9be7-dd23ddae08af", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "380f80446c2da05db55073c18ea36c24e99f1a3b09e2ebfe6cf77a182645b4db", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6127e185-99c8-454b-b67d-6ed790f48d8b", "node_type": "1", "metadata": {}, "hash": "332e480efc41cae58485573f76a0402fbc065de4b38e5b72408088e6d4f58cff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 858, "end_char_idx": 3383, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Types of Segmentation

Semantic segmentation assigns a label or class to every single pixel in an image. Let's take a look at a semantic segmentation model output. It will assign the same class to every instance of an object it comes across in an image, for example, all cats will be labeled as "cat" instead of "cat-1", "cat-2".
We can use transformers' image segmentation pipeline to quickly infer a semantic segmentation model. Let's take a look at the example image.

```python
from transformers import pipeline
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg" alt="Segmentation Input"/>
</div>

We will use [nvidia/segformer-b1-finetuned-cityscapes-1024-1024](https://huggingface.co/nvidia/segformer-b1-finetuned-cityscapes-1024-1024).

```python
semantic_segmentation = pipeline("image-segmentation", "nvidia/segformer-b1-finetuned-cityscapes-1024-1024")
results = semantic_segmentation(image)
results
```

The segmentation pipeline output includes a mask for every predicted class.
```bash
[{'score': None,
  'label': 'road',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sidewalk',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'building',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'wall',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'pole',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'traffic sign',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'vegetation',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'terrain',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```

Taking a look at the mask for the car class, we can see every car is classified with the same mask.


key: _node_content
string_value:
{"id_": "6127e185-99c8-454b-b67d-6ed790f48d8b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "356e94a0-c248-4a1b-a2c8-966c554443f4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "4c2904b57466cfab2ca0f2ff0976afa78211285f893067aa1eaaac15d6f2093d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "39d9332c-d4a6-4a13-9d5a-8c16ab3aac1d", "node_type": "1", "metadata": {}, "hash": "d19353615ce3434092b174583dda05256f54f985c7c13b5ac4fbb23fc4319713", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2328, "end_char_idx": 4836, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
[{'score': None,
  'label': 'road',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sidewalk',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'building',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'wall',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'pole',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'traffic sign',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'vegetation',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'terrain',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```

Taking a look at the mask for the car class, we can see every car is classified with the same mask.

```python
results[-1]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/semantic_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

In instance segmentation, the goal is not to classify every pixel, but to predict a mask for **every instance of an object** in a given image. It works very similar to object detection, where there is a bounding box for every instance, there's a segmentation mask instead. We will use [facebook/mask2former-swin-large-cityscapes-instance](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-instance) for this.

```python
instance_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-instance")
results = instance_segmentation(image)
results
```

As you can see below, there are multiple cars classified, and there's no classification for pixels other than pixels that belong to car and person instances.

```bash
[{'score': 0.999944,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999945,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999652,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.903529,
  'label': 'person',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```
Checking out one of the car masks below.


key: _node_content
string_value:
{"id_": "39d9332c-d4a6-4a13-9d5a-8c16ab3aac1d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6127e185-99c8-454b-b67d-6ed790f48d8b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "6c717e9d2becbb67f34a94ca03018257d066f3527f92762c70e44ca2d5fd5bc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e57d6dac-8eb2-4b14-905b-c6953514f79f", "node_type": "1", "metadata": {}, "hash": "e48832e7c3c106d2f31feb8510cce7b1d819460b8dfeda004749a8a783d24f53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3385, "end_char_idx": 5765, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results[-1]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/semantic_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

In instance segmentation, the goal is not to classify every pixel, but to predict a mask for **every instance of an object** in a given image. It works very similar to object detection, where there is a bounding box for every instance, there's a segmentation mask instead. We will use [facebook/mask2former-swin-large-cityscapes-instance](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-instance) for this.

```python
instance_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-instance")
results = instance_segmentation(image)
results
```

As you can see below, there are multiple cars classified, and there's no classification for pixels other than pixels that belong to car and person instances.

```bash
[{'score': 0.999944,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999945,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999652,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.903529,
  'label': 'person',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```
Checking out one of the car masks below.

```python
results[2]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/instance_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

Panoptic segmentation combines semantic segmentation and instance segmentation, where every pixel is classified into a class and an instance of that class, and there are multiple masks for each instance of a class. We can use [facebook/mask2former-swin-large-cityscapes-panoptic](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-panoptic) for this.

```python
panoptic_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-panoptic")
results = panoptic_segmentation(image)
results
```
As you can see below, we have more classes. We will later illustrate to see that every pixel is classified into one of the classes.


key: _node_content
string_value:
{"id_": "e57d6dac-8eb2-4b14-905b-c6953514f79f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "39d9332c-d4a6-4a13-9d5a-8c16ab3aac1d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "45cff402a5613421842f40ea9bfa0006d8c79714b397d3584a11529866a5ee89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1daa9fd2-5ea7-4fb1-907c-a5a9399f82ad", "node_type": "1", "metadata": {}, "hash": "27b255274b441150644516f516d93eea7eb1c96885cdeb37882b514bd088ccc1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4401, "end_char_idx": 7045, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
[{'score': 0.999944,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999945,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999652,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.903529,
  'label': 'person',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```
Checking out one of the car masks below.

```python
results[2]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/instance_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

Panoptic segmentation combines semantic segmentation and instance segmentation, where every pixel is classified into a class and an instance of that class, and there are multiple masks for each instance of a class. We can use [facebook/mask2former-swin-large-cityscapes-panoptic](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-panoptic) for this.

```python
panoptic_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-panoptic")
results = panoptic_segmentation(image)
results
```
As you can see below, we have more classes. We will later illustrate to see that every pixel is classified into one of the classes.

```bash
[{'score': 0.999981,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999958,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.99997,
  'label': 'vegetation',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999575,
  'label': 'pole',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999958,
  'label': 'building',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999634,
  'label': 'road',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.996092,
  'label': 'sidewalk',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999221,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.99987,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```

Let's have a side by side comparison for all types of segmentation.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation-comparison.png" alt="Segmentation Maps Compared"/>
</div>

Seeing all types of segmentation, let's have a deep dive on fine-tuning a model for semantic segmentation.


key: _node_content
string_value:
{"id_": "1daa9fd2-5ea7-4fb1-907c-a5a9399f82ad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e57d6dac-8eb2-4b14-905b-c6953514f79f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "aca7de62a6857bf1dc289222936c2e61ce26ac02b46ffe420b09c0f0daadc5cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45eac192-9ac7-4888-9a1b-3ea627dbdb63", "node_type": "1", "metadata": {}, "hash": "889e3b3d6625a79644a3da7340abdbaa71aa3916a57d49cd34bb75056a3950e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6721, "end_char_idx": 9110, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation-comparison.png" alt="Segmentation Maps Compared"/>
</div>

Seeing all types of segmentation, let's have a deep dive on fine-tuning a model for semantic segmentation.

Common real-world applications of semantic segmentation include training self-driving cars to identify pedestrians and important traffic information, identifying cells and abnormalities in medical imagery, and monitoring environmental changes from satellite imagery.

## Fine-tuning a Model for Segmentation

We will now:

1. Finetune [SegFormer](https://huggingface.co/docs/transformers/main/en/model_doc/segformer#segformer) on the [SceneParse150](https://huggingface.co/datasets/scene_parse_150) dataset.
2. Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-segmentation)

</Tip>


### Load SceneParse150 dataset

Start by loading a smaller subset of the SceneParse150 dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> ds = ds.train_test_split(test_size=0.2)
>>> train_ds = ds["train"]
>>> test_ds = ds["test"]
```

Then take a look at an example:

```py
>>> train_ds[0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,
 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,
 'scene_category': 368}

# view the image
>>> train_ds[0]["image"]
```

- `image`: a PIL image of the scene.
- `annotation`: a PIL image of the segmentation map, which is also the model's target.
- `scene_category`: a category id that describes the image scene like "kitchen" or "office". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.

You'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later.


key: _node_content
string_value:
{"id_": "45eac192-9ac7-4888-9a1b-3ea627dbdb63", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1daa9fd2-5ea7-4fb1-907c-a5a9399f82ad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "b9c15964aab212a3fe6cc9f7405eab099bceffc625fbdb572c8a2512cabd3b1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dab2c989-d18a-41f0-96c9-8f9d8db96e22", "node_type": "1", "metadata": {}, "hash": "1f9d8ee40e2d2fbf13dd9983ec089b2be97e662ce07b844c9119aa24aba5ec27", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7600, "end_char_idx": 9916, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-segmentation)

</Tip>


### Load SceneParse150 dataset

Start by loading a smaller subset of the SceneParse150 dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> ds = ds.train_test_split(test_size=0.2)
>>> train_ds = ds["train"]
>>> test_ds = ds["test"]
```

Then take a look at an example:

```py
>>> train_ds[0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,
 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,
 'scene_category': 368}

# view the image
>>> train_ds[0]["image"]
```

- `image`: a PIL image of the scene.
- `annotation`: a PIL image of the segmentation map, which is also the model's target.
- `scene_category`: a category id that describes the image scene like "kitchen" or "office". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.

You'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the `id2label` and `label2id` dictionaries:

```py
>>> import json
>>> from pathlib import Path
>>> from huggingface_hub import hf_hub_download

>>> repo_id = "huggingface/label-files"
>>> filename = "ade20k-id2label.json"
>>> id2label = json.loads(Path(hf_hub_download(repo_id, filename, repo_type="dataset")).read_text())
>>> id2label = {int(k): v for k, v in id2label.items()}
>>> label2id = {v: k for k, v in id2label.items()}
>>> num_labels = len(id2label)
```

#### Custom dataset

You could also create and use your own dataset if you prefer to train with the [run_semantic_segmentation.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py) script instead of a notebook instance.


key: _node_content
string_value:
{"id_": "dab2c989-d18a-41f0-96c9-8f9d8db96e22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45eac192-9ac7-4888-9a1b-3ea627dbdb63", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "23df824c7dc5a6f1e204cef5673e98e43ad2205ed3581e47c395d86d07af55f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0689bd6-bdc6-4074-af16-d536a0febdf7", "node_type": "1", "metadata": {}, "hash": "42d94ed0b47c326fd19a6ff0cea5f0ad6679e4b4386896470be65edb34dbec02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8710, "end_char_idx": 11901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `annotation`: a PIL image of the segmentation map, which is also the model's target.
- `scene_category`: a category id that describes the image scene like "kitchen" or "office". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.

You'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the `id2label` and `label2id` dictionaries:

```py
>>> import json
>>> from pathlib import Path
>>> from huggingface_hub import hf_hub_download

>>> repo_id = "huggingface/label-files"
>>> filename = "ade20k-id2label.json"
>>> id2label = json.loads(Path(hf_hub_download(repo_id, filename, repo_type="dataset")).read_text())
>>> id2label = {int(k): v for k, v in id2label.items()}
>>> label2id = {v: k for k, v in id2label.items()}
>>> num_labels = len(id2label)
```

#### Custom dataset

You could also create and use your own dataset if you prefer to train with the [run_semantic_segmentation.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py) script instead of a notebook instance. The script requires:

1. a [`~datasets.DatasetDict`] with two [`~datasets.Image`] columns, "image" and "label"

     ```py
     from datasets import Dataset, DatasetDict, Image

     image_paths_train = ["path/to/image_1.jpg/jpg", "path/to/image_2.jpg/jpg", ..., "path/to/image_n.jpg/jpg"]
     label_paths_train = ["path/to/annotation_1.png", "path/to/annotation_2.png", ..., "path/to/annotation_n.png"]

     image_paths_validation = [...]
     label_paths_validation = [...]

     def create_dataset(image_paths, label_paths):
         dataset = Dataset.from_dict({"image": sorted(image_paths),
                                     "label": sorted(label_paths)})
         dataset = dataset.cast_column("image", Image())
         dataset = dataset.cast_column("label", Image())
         return dataset

     # step 1: create Dataset objects
     train_dataset = create_dataset(image_paths_train, label_paths_train)
     validation_dataset = create_dataset(image_paths_validation, label_paths_validation)

     # step 2: create DatasetDict
     dataset = DatasetDict({
          "train": train_dataset,
          "validation": validation_dataset,
          }
     )

     # step 3: push to Hub (assumes you have ran the huggingface-cli login command in a terminal/notebook)
     dataset.push_to_hub("your-name/dataset-repo")

     # optionally, you can push to a private repo on the Hub
     # dataset.push_to_hub("name of repo on the hub", private=True)
     ```

2. an id2label dictionary mapping the class integers to their class names

     ```py
     import json
     # simple example
     id2label = {0: 'cat', 1: 'dog'}
     with open('id2label.json', 'w') as fp:
     json.dump(id2label, fp)
     ```

As an example, take a look at this [example dataset](https://huggingface.co/datasets/nielsr/ade20k-demo) which was created with the steps shown above.

### Preprocess

The next step is to load a SegFormer image processor to prepare the images and annotations for the model.


key: _node_content
string_value:
{"id_": "e0689bd6-bdc6-4074-af16-d536a0febdf7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dab2c989-d18a-41f0-96c9-8f9d8db96e22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "d46664c59773f62929df690c94ef508ca404dc3d359501d19af0e5ef623dd662", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3992dfcd-dbe9-4d62-8b18-a1e997b74330", "node_type": "1", "metadata": {}, "hash": "0c9d533a75316f71ec98e4ecc3c9aa81d70fea132a66dc3e87d91e1ed3be6646", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11780, "end_char_idx": 14827, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Preprocess

The next step is to load a SegFormer image processor to prepare the images and annotations for the model. Some datasets, like this one, use the zero-index as the background class. However, the background class isn't actually included in the 150 classes, so you'll need to set `do_reduce_labels=True` to subtract one from all the labels. The zero-index is replaced by `255` so it's ignored by SegFormer's loss function:

```py
>>> from transformers import AutoImageProcessor

>>> checkpoint = "nvidia/mit-b0"
>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint, do_reduce_labels=True)
```

<frameworkcontent>
<pt>

It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting. In this guide, you'll use the [`ColorJitter`](https://pytorch.org/vision/stable/generated/torchvision.transforms.ColorJitter.html) function from [torchvision](https://pytorch.org/vision/stable/index.html) to randomly change the color properties of an image, but you can also use any image library you like.

```py
>>> from torchvision.transforms import ColorJitter

>>> jitter = ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.1)
```

Now create two preprocessing functions to prepare the images and annotations for the model. These functions convert the images into `pixel_values` and annotations to `labels`. For the training set, `jitter` is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the `images`, and only crops the `labels` because no data augmentation is applied during testing.

```py
>>> def train_transforms(example_batch):
...     images = [jitter(x) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


>>> def val_transforms(example_batch):
...     images = [x for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
```

To apply the `jitter` over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.set_transform`] function. The transform is applied on the fly which is faster and consumes less disk space:

```py
>>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.
In this guide, you'll use [`tf.image`](https://www.tensorflow.org/api_docs/python/tf/image) to randomly change the color properties of an image, but you can also use any image
library you like.
Define two separate transformation functions:
- training data transformations that include image augmentation
- validation data transformations that only transpose the images, since computer vision models in 🤗 Transformers expect channels-first layout

```py
>>> import tensorflow as tf


key: _node_content
string_value:
{"id_": "3992dfcd-dbe9-4d62-8b18-a1e997b74330", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0689bd6-bdc6-4074-af16-d536a0febdf7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "4fd5c37146496b7ba18de909bf92906a8e570a33136fd4865571be0cb87b41ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca19de49-58f1-4c8c-abee-360febd76628", "node_type": "1", "metadata": {}, "hash": "32889705aca16e332ef49ef69a1e05e88bdfef77fdf4ac66a6fa3b27f58b8bb9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13087, "end_char_idx": 15955, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These functions convert the images into `pixel_values` and annotations to `labels`. For the training set, `jitter` is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the `images`, and only crops the `labels` because no data augmentation is applied during testing.

```py
>>> def train_transforms(example_batch):
...     images = [jitter(x) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


>>> def val_transforms(example_batch):
...     images = [x for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
```

To apply the `jitter` over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.set_transform`] function. The transform is applied on the fly which is faster and consumes less disk space:

```py
>>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.
In this guide, you'll use [`tf.image`](https://www.tensorflow.org/api_docs/python/tf/image) to randomly change the color properties of an image, but you can also use any image
library you like.
Define two separate transformation functions:
- training data transformations that include image augmentation
- validation data transformations that only transpose the images, since computer vision models in 🤗 Transformers expect channels-first layout

```py
>>> import tensorflow as tf


>>> def aug_transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.image.random_brightness(image, 0.25)
...     image = tf.image.random_contrast(image, 0.5, 2.0)
...     image = tf.image.random_saturation(image, 0.75, 1.25)
...     image = tf.image.random_hue(image, 0.1)
...     image = tf.transpose(image, (2, 0, 1))
...     return image


>>> def transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.transpose(image, (2, 0, 1))
...     return image
```

Next, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply
the image transformations and use the earlier loaded `image_processor` to convert the images into `pixel_values` and
annotations to `labels`. `ImageProcessor` also takes care of resizing and normalizing the images.

```py
>>> def train_transforms(example_batch):
...     images = [aug_transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


key: _node_content
string_value:
{"id_": "ca19de49-58f1-4c8c-abee-360febd76628", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3992dfcd-dbe9-4d62-8b18-a1e997b74330", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "bc877763f8194c7770650f04565a975770ccd8d30a9993b8b5486603726e0a5a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c4ab1ce-e0dd-4102-92f1-51283ff05e59", "node_type": "1", "metadata": {}, "hash": "f64919519d54ee46dcc789813149f5d5cf857cdbd88e3dde70f071cc4d69c7f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14830, "end_char_idx": 18339, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def aug_transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.image.random_brightness(image, 0.25)
...     image = tf.image.random_contrast(image, 0.5, 2.0)
...     image = tf.image.random_saturation(image, 0.75, 1.25)
...     image = tf.image.random_hue(image, 0.1)
...     image = tf.transpose(image, (2, 0, 1))
...     return image


>>> def transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.transpose(image, (2, 0, 1))
...     return image
```

Next, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply
the image transformations and use the earlier loaded `image_processor` to convert the images into `pixel_values` and
annotations to `labels`. `ImageProcessor` also takes care of resizing and normalizing the images.

```py
>>> def train_transforms(example_batch):
...     images = [aug_transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


>>> def val_transforms(example_batch):
...     images = [transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
```

To apply the preprocessing transformations over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.set_transform`] function.
The transform is applied on the fly which is faster and consumes less disk space:

```py
>>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
```
</tf>
</frameworkcontent>

### Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load an evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [mean Intersection over Union](https://huggingface.co/spaces/evaluate-metric/accuracy) (IoU) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> metric = evaluate.load("mean_iou")
```

Then create a function to [`~evaluate.EvaluationModule.compute`] the metrics. Your predictions need to be converted to
logits first, and then reshaped to match the size of the labels before you can call [`~evaluate.EvaluationModule.compute`]:

<frameworkcontent>
<pt>

```py
>>> import numpy as np
>>> import torch
>>> from torch import nn

>>> def compute_metrics(eval_pred):
...     with torch.no_grad():
...         logits, labels = eval_pred
...         logits_tensor = torch.from_numpy(logits)
...         logits_tensor = nn.functional.interpolate(
...             logits_tensor,
...             size=labels.shape[-2:],
...             mode="bilinear",
...             align_corners=False,
...         ).argmax(dim=1)

...         pred_labels = logits_tensor.detach().cpu().numpy()
...         metrics = metric.compute(
...             predictions=pred_labels,
...             references=labels,
...             num_labels=num_labels,
...             ignore_index=255,
...             reduce_labels=False,
...         )
...         for key, value in metrics.items():
...             if isinstance(value, np.ndarray):
...                 metrics[key] = value.tolist()
...         return metrics
```

</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8c4ab1ce-e0dd-4102-92f1-51283ff05e59", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca19de49-58f1-4c8c-abee-360febd76628", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "0d8d210065e19f087ad642e76cefdac009d92fdb5628fc367491f8e8e9650d32", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79c5886d-4536-4484-891e-c73548f87893", "node_type": "1", "metadata": {}, "hash": "ee45fd45fe330e33ea671d3741c48858a1725e958bad49742727131c00d3fac7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18342, "end_char_idx": 20884, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<tf>

```py
>>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     logits = tf.transpose(logits, perm=[0, 2, 3, 1])
...     logits_resized = tf.image.resize(
...         logits,
...         size=tf.shape(labels)[1:],
...         method="bilinear",
...     )

...     pred_labels = tf.argmax(logits_resized, axis=-1)
...     metrics = metric.compute(
...         predictions=pred_labels,
...         references=labels,
...         num_labels=num_labels,
...         ignore_index=-1,
...         reduce_labels=image_processor.do_reduce_labels,
...     )

...     per_category_accuracy = metrics.pop("per_category_accuracy").tolist()
...     per_category_iou = metrics.pop("per_category_iou").tolist()

...     metrics.update({f"accuracy_{id2label[i]}": v for i, v in enumerate(per_category_accuracy)})
...     metrics.update({f"iou_{id2label[i]}": v for i, v in enumerate(per_category_iou)})
...     return {"val_" + k: v for k, v in metrics.items()}
```

</tf>
</frameworkcontent>

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

### Train
<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#finetune-with-trainer)!

</Tip>

You're ready to start training your model now! Load SegFormer with [`AutoModelForSemanticSegmentation`], and pass the model the mapping between label ids and label classes:

```py
>>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer

>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because this'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the IoU metric and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "79c5886d-4536-4484-891e-c73548f87893", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c4ab1ce-e0dd-4102-92f1-51283ff05e59", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "a95218c59b575483998747eec896048801c453a841608f524756e18d35a914ec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "197e4f25-1581-4857-8bdd-c5c12abf43f5", "node_type": "1", "metadata": {}, "hash": "99650b7cf77a92208f6a77c57a65980fa003762a9a1fe65c26224f0e021c9099", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19473, "end_char_idx": 22350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Train
<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#finetune-with-trainer)!

</Tip>

You're ready to start training your model now! Load SegFormer with [`AutoModelForSemanticSegmentation`], and pass the model the mapping between label ids and label classes:

```py
>>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer

>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because this'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the IoU metric and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="segformer-b0-scene-parse-150",
...     learning_rate=6e-5,
...     num_train_epochs=50,
...     per_device_train_batch_size=2,
...     per_device_eval_batch_size=2,
...     save_total_limit=3,
...     eval_strategy="steps",
...     save_strategy="steps",
...     save_steps=20,
...     eval_steps=20,
...     logging_steps=1,
...     eval_accumulation_steps=5,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=train_ds,
...     eval_dataset=test_ds,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pretrained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks to calculate metrics and upload your model to 🤗 Hub
6. Use the `fit()` method to run the training.


key: _node_content
string_value:
{"id_": "197e4f25-1581-4857-8bdd-c5c12abf43f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79c5886d-4536-4484-891e-c73548f87893", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "2c502e3ca533d04c41ad666d171551e018c13a8c8ae8278659854711f59823be", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "498222bc-814f-4be9-8630-9520c827f15e", "node_type": "1", "metadata": {}, "hash": "7029ad31362defd4ebe54fe57f2d3d563e0cab9abf5c0c5e242e680f5e19f722", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20834, "end_char_idx": 23366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="segformer-b0-scene-parse-150",
...     learning_rate=6e-5,
...     num_train_epochs=50,
...     per_device_train_batch_size=2,
...     per_device_eval_batch_size=2,
...     save_total_limit=3,
...     eval_strategy="steps",
...     save_strategy="steps",
...     save_steps=20,
...     eval_steps=20,
...     logging_steps=1,
...     eval_accumulation_steps=5,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=train_ds,
...     eval_dataset=test_ds,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pretrained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks to calculate metrics and upload your model to 🤗 Hub
6. Use the `fit()` method to run the training.

Start by defining the hyperparameters, optimizer and learning rate schedule:

```py
>>> from transformers import create_optimizer

>>> batch_size = 2
>>> num_epochs = 50
>>> num_train_steps = len(train_ds) * num_epochs
>>> learning_rate = 6e-5
>>> weight_decay_rate = 0.01

>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... )
```

Then, load SegFormer with [`TFAutoModelForSemanticSegmentation`] along with the label mappings, and compile it with the
optimizer. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
... )
>>> model.compile(optimizer=optimizer)  # No loss argument!


key: _node_content
string_value:
{"id_": "498222bc-814f-4be9-8630-9520c827f15e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "197e4f25-1581-4857-8bdd-c5c12abf43f5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "9cd31ce7f2d3930406764d516aee4b6effe8f6399f119e798cec836ce280790c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ebe5476-1cb6-465c-9563-a8060530d330", "node_type": "1", "metadata": {}, "hash": "9a49d4c667aab8e8cff7116217554b6f2008b61206e7c039915cb2d739b64817", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21972, "end_char_idx": 25213, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pretrained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks to calculate metrics and upload your model to 🤗 Hub
6. Use the `fit()` method to run the training.

Start by defining the hyperparameters, optimizer and learning rate schedule:

```py
>>> from transformers import create_optimizer

>>> batch_size = 2
>>> num_epochs = 50
>>> num_train_steps = len(train_ds) * num_epochs
>>> learning_rate = 6e-5
>>> weight_decay_rate = 0.01

>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... )
```

Then, load SegFormer with [`TFAutoModelForSemanticSegmentation`] along with the label mappings, and compile it with the
optimizer. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
... )
>>> model.compile(optimizer=optimizer)  # No loss argument!
```

Convert your datasets to the `tf.data.Dataset` format using the [`~datasets.Dataset.to_tf_dataset`] and the [`DefaultDataCollator`]:

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator(return_tensors="tf")

>>> tf_train_dataset = train_ds.to_tf_dataset(
...     columns=["pixel_values", "label"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )

>>> tf_eval_dataset = test_ds.to_tf_dataset(
...     columns=["pixel_values", "label"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
```

To compute the accuracy from the predictions and push your model to the 🤗 Hub, use [Keras callbacks](../main_classes/keras_callbacks).
Pass your `compute_metrics` function to [`KerasMetricCallback`],
and use the [`PushToHubCallback`] to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )

>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)

>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... )
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>

### Inference

Great, now that you've finetuned a model, you can use it for inference!


key: _node_content
string_value:
{"id_": "0ebe5476-1cb6-465c-9563-a8060530d330", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "498222bc-814f-4be9-8630-9520c827f15e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "41bf2b623ce6c543c91ffcc3a990017a435d1bd6754c1e37b8cacb3079cbc658", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fbdf6b0-6533-4233-9410-eb8c94d5d897", "node_type": "1", "metadata": {}, "hash": "7b6c332f760af9acedcc19b651959283d44cfbfdb2f81cdbc404410fce0a7921", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24129, "end_char_idx": 25749, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass your `compute_metrics` function to [`KerasMetricCallback`],
and use the [`PushToHubCallback`] to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )

>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)

>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... )
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>

### Inference

Great, now that you've finetuned a model, you can use it for inference!

Reload the dataset and load an image for inference.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
>>> ds = ds.train_test_split(test_size=0.2)
>>> test_ds = ds["test"]
>>> image = ds["test"][0]["image"]
>>> image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png" alt="Image of bedroom"/>
</div>

<frameworkcontent>
<pt>

We will now see how to infer without a pipeline.


