# embedding_metadata

key: chroma:document
string_value:
```py
>>> import torch

>>> from transformers import IdeficsForVisionText2Text, AutoProcessor

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Setting `device_map` to `"auto"` will automatically determine how to load and store the model weights in the most optimized 
manner given existing devices.

### Quantized model

If high-memory GPU availability is an issue, you can load the quantized version of the model. To load the model and the 
processor in 4bit precision, pass a `BitsAndBytesConfig` to the `from_pretrained` method and the model will be compressed 
on the fly while loading.

```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig

>>> quantization_config = BitsAndBytesConfig(
...     load_in_4bit=True,
...     bnb_4bit_compute_dtype=torch.float16,
... )

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(
...     checkpoint,
...     quantization_config=quantization_config,
...     device_map="auto"
... )
```

Now that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.

## Image captioning
Image captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired 
people navigate through different situations, for instance, explore image content online. 

To illustrate the task, get an image to be captioned, e.g.:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-im-captioning.jpg" alt="Image of a puppy in a flower bed"/>
</div>

Photo by [Hendo Wang](https://unsplash.com/@hendoo). 

IDEFICS accepts text and image prompts. However, to caption an image, you do not have to provide a text prompt to the 
model, only the preprocessed input image. Without a text prompt, the model will start generating text from the 
BOS (beginning-of-sequence) token thus creating a caption.

As image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.


key: _node_content
string_value:
{"id_": "419af2bb-b4d9-4000-85b2-c0ae8c67d43f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "242a285b-537a-46d4-80ca-2276c3dc15cc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "5b4f5570c180f7bf576e64b5461c69788eb727d15f0b2b85151b95c023ab0b7d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ea25d55-fa0f-40e7-bc84-3e228eb14587", "node_type": "1", "metadata": {}, "hash": "65ea3011d628dee36c517d306deb9deeaf604bef2ac68d605ca3af5917cb66a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3789, "end_char_idx": 6581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig

>>> quantization_config = BitsAndBytesConfig(
...     load_in_4bit=True,
...     bnb_4bit_compute_dtype=torch.float16,
... )

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(
...     checkpoint,
...     quantization_config=quantization_config,
...     device_map="auto"
... )
```

Now that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.

## Image captioning
Image captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired 
people navigate through different situations, for instance, explore image content online. 

To illustrate the task, get an image to be captioned, e.g.:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-im-captioning.jpg" alt="Image of a puppy in a flower bed"/>
</div>

Photo by [Hendo Wang](https://unsplash.com/@hendoo). 

IDEFICS accepts text and image prompts. However, to caption an image, you do not have to provide a text prompt to the 
model, only the preprocessed input image. Without a text prompt, the model will start generating text from the 
BOS (beginning-of-sequence) token thus creating a caption.

As image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3542&q=80",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
A puppy in a flower bed
```

<Tip>

It is a good idea to include the `bad_words_ids` in the call to `generate` to avoid errors arising when increasing 
the `max_new_tokens`: the model will want to generate a new `<image>` or `<fake_token_around_image>` token when there 
is no image being generated by the model.
You can set it on-the-fly as in this guide, or store in the `GenerationConfig` as described in the [Text generation strategies](../generation_strategies) guide.
</Tip>

## Prompted image captioning

You can extend image captioning by providing a text prompt, which the model will continue given the image.


key: _node_content
string_value:
{"id_": "5ea25d55-fa0f-40e7-bc84-3e228eb14587", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "419af2bb-b4d9-4000-85b2-c0ae8c67d43f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "8b21268d119b02729689bc0fee05003f69fe679e255c0c1cac8f5ea07c3c3daa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c", "node_type": "1", "metadata": {}, "hash": "654548ee4c21d47b1c2af7c733295e93ef9e27c3a9323b1a09b738509de0df7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5111, "end_char_idx": 7991, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Without a text prompt, the model will start generating text from the 
BOS (beginning-of-sequence) token thus creating a caption.

As image input to the model, you can use either an image object (`PIL.Image`) or a url from which the image can be retrieved.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3542&q=80",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
A puppy in a flower bed
```

<Tip>

It is a good idea to include the `bad_words_ids` in the call to `generate` to avoid errors arising when increasing 
the `max_new_tokens`: the model will want to generate a new `<image>` or `<fake_token_around_image>` token when there 
is no image being generated by the model.
You can set it on-the-fly as in this guide, or store in the `GenerationConfig` as described in the [Text generation strategies](../generation_strategies) guide.
</Tip>

## Prompted image captioning

You can extend image captioning by providing a text prompt, which the model will continue given the image. Let's take 
another image to illustrate:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-prompted-im-captioning.jpg" alt="Image of the Eiffel Tower at night"/>
</div>

Photo by [Denys Nevozhai](https://unsplash.com/@dnevozhai).
   
Textual and image prompts can be passed to the model's processor as a single list to create appropriate inputs.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...     "This is an image of ",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
This is an image of the Eiffel Tower in Paris, France.
```

## Few-shot prompting

While IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with 
other restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.


key: _node_content
string_value:
{"id_": "88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ea25d55-fa0f-40e7-bc84-3e228eb14587", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "fa539d43a77412145ceab596d8890de2036b120262a1fa2bb51ad0eacce7e8c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9765e6c5-dd4c-4977-867c-d0f3f7ab9232", "node_type": "1", "metadata": {}, "hash": "da22b7ba743231e97503e1c29f8db4dea794aabfdcdcc7948f3d4cd4e7218d15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6582, "end_char_idx": 9041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's take 
another image to illustrate:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-prompted-im-captioning.jpg" alt="Image of the Eiffel Tower at night"/>
</div>

Photo by [Denys Nevozhai](https://unsplash.com/@dnevozhai).
   
Textual and image prompts can be passed to the model's processor as a single list to create appropriate inputs.

```py
>>> prompt = [
...     "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...     "This is an image of ",
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
This is an image of the Eiffel Tower in Paris, France.
```

## Few-shot prompting

While IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with 
other restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.
By providing examples in the prompt, you can steer the model to generate results that mimic the format of given examples. 

Let's use the previous image of the Eiffel Tower as an example for the model and build a prompt that demonstrates to the model 
that in addition to learning what the object in an image is, we would also like to get some interesting information about it. 
Then, let's see, if we can get the same response format for an image of the Statue of Liberty:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg" alt="Image of the Statue of Liberty"/>
</div>

Photo by [Juan Mayobre](https://unsplash.com/@jmayobres).
  
```py
>>> prompt = ["User:",
...            "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...            "Describe this image.\nAssistant: An image of the Eiffel Tower at night.


key: _node_content
string_value:
{"id_": "9765e6c5-dd4c-4977-867c-d0f3f7ab9232", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "88a4bb26-1cf9-4c89-8339-1aa57dc1dd6c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "44e1da43cb969f2d6fa55a275cae6f6acbbe691e613624877436973cc4b988f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79cbe94a-a98e-47f6-8cab-e97329935038", "node_type": "1", "metadata": {}, "hash": "8ab4a099d58b7dd269830a0ff351a502f2e7bd7ef074339997e740a7193538d3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7713, "end_char_idx": 10618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

## Few-shot prompting

While IDEFICS demonstrates great zero-shot results, your task may require a certain format of the caption, or come with 
other restrictions or requirements that increase task's complexity. Few-shot prompting can be used to enable in-context learning.
By providing examples in the prompt, you can steer the model to generate results that mimic the format of given examples. 

Let's use the previous image of the Eiffel Tower as an example for the model and build a prompt that demonstrates to the model 
that in addition to learning what the object in an image is, we would also like to get some interesting information about it. 
Then, let's see, if we can get the same response format for an image of the Statue of Liberty:

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg" alt="Image of the Statue of Liberty"/>
</div>

Photo by [Juan Mayobre](https://unsplash.com/@jmayobres).
  
```py
>>> prompt = ["User:",
...            "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...            "Describe this image.\nAssistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building.\n",
...            "User:",
...            "https://images.unsplash.com/photo-1524099163253-32b7f0256868?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3387&q=80",
...            "Describe this image.\nAssistant:"
...            ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=30, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
User: Describe this image.
Assistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. 
User: Describe this image.
Assistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.
```

Notice that just from a single example (i.e., 1-shot) the model has learned how to perform the task. For more complex tasks, 
feel free to experiment with a larger number of examples (e.g., 3-shot, 5-shot, etc.).

## Visual question answering

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image. Similar to image 
captioning it can be used in accessibility applications, but also in education (reasoning about visual materials), customer 
service (questions about products based on images), and image retrieval.


key: _node_content
string_value:
{"id_": "79cbe94a-a98e-47f6-8cab-e97329935038", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9765e6c5-dd4c-4977-867c-d0f3f7ab9232", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "b5fdca2836360a87e76a373fd7328ee9774efd6b7cf9ef04a9ef3697c3ffcd3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95361241-f724-462a-b6f0-1bbbccc4b3ca", "node_type": "1", "metadata": {}, "hash": "c8f66bfa86885f3915a179717b6d0338d4942e643b5475e337780cba910cd25b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9811, "end_char_idx": 12703, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Assistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. 
User: Describe this image.
Assistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.
```

Notice that just from a single example (i.e., 1-shot) the model has learned how to perform the task. For more complex tasks, 
feel free to experiment with a larger number of examples (e.g., 3-shot, 5-shot, etc.).

## Visual question answering

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image. Similar to image 
captioning it can be used in accessibility applications, but also in education (reasoning about visual materials), customer 
service (questions about products based on images), and image retrieval.

Let's get a new image for this task: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-vqa.jpg" alt="Image of a couple having a picnic"/>
</div>

Photo by [Jarritos Mexican Soda](https://unsplash.com/@jarritos). 

You can steer the model from image captioning to visual question answering by prompting it with appropriate instructions: 

```py
>>> prompt = [
...     "Instruction: Provide an answer to the question. Use the image to answer.\n",
...     "https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",
...     "Question: Where are these people and what's the weather like? Answer:"
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=20, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
Instruction: Provide an answer to the question. Use the image to answer.
 Question: Where are these people and what's the weather like? Answer: They're in a park in New York City, and it's a beautiful day.
```

## Image classification

IDEFICS is capable of classifying images into different categories without being explicitly trained on data containing 
labeled examples from those specific categories. Given a list of categories and using its image and text understanding 
capabilities, the model can infer which category the image likely belongs to. 

Say, we have this image of a vegetable stand: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-classification.jpg" alt="Image of a vegetable stand"/>
</div>

Photo by [Peter Wendt](https://unsplash.com/@peterwendt).


key: _node_content
string_value:
{"id_": "95361241-f724-462a-b6f0-1bbbccc4b3ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79cbe94a-a98e-47f6-8cab-e97329935038", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "89c58d9b593d46fcebc83a01668db25b75b2a65689be2029916086dc24b2e06d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6fda5281-4693-4f63-9332-ec5f94eb56ad", "node_type": "1", "metadata": {}, "hash": "20c5eabf06d276d9cda6a51d857c804a7c0d6e1057a6c11ff9896d4510ae0363", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11420, "end_char_idx": 14556, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Answer:"
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=20, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
Instruction: Provide an answer to the question. Use the image to answer.
 Question: Where are these people and what's the weather like? Answer: They're in a park in New York City, and it's a beautiful day.
```

## Image classification

IDEFICS is capable of classifying images into different categories without being explicitly trained on data containing 
labeled examples from those specific categories. Given a list of categories and using its image and text understanding 
capabilities, the model can infer which category the image likely belongs to. 

Say, we have this image of a vegetable stand: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-classification.jpg" alt="Image of a vegetable stand"/>
</div>

Photo by [Peter Wendt](https://unsplash.com/@peterwendt).

We can instruct the model to classify the image into one of the categories that we have:

```py
>>> categories = ['animals','vegetables', 'city landscape', 'cars', 'office']
>>> prompt = [f"Instruction: Classify the following image into a single category from the following list: {categories}.\n",
...     "https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",    
...     "Category: "
... ]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=6, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0])
Instruction: Classify the following image into a single category from the following list: ['animals', 'vegetables', 'city landscape', 'cars', 'office'].
Category: Vegetables
```  

In the example above we instruct the model to classify the image into a single category, however, you can also prompt the model to do rank classification.

## Image-guided text generation

For more creative applications, you can use image-guided text generation to generate text based on an image. This can be 
useful to create descriptions of products, ads, descriptions of a scene, etc. 

Let's prompt IDEFICS to write a story based on a simple image of a red door: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-story-generation.jpg" alt="Image of a red door with a pumpkin on the steps"/>
</div>

Photo by [Craig Tidball](https://unsplash.com/@devonshiremedia).


key: _node_content
string_value:
{"id_": "6fda5281-4693-4f63-9332-ec5f94eb56ad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95361241-f724-462a-b6f0-1bbbccc4b3ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "d23047bac1019360c1aeec82f0d48ac8cd687c761c4198ce78551b8b84152fd8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "36d6d607-3d26-4d35-9762-e16ebdab0351", "node_type": "1", "metadata": {}, "hash": "76fcaeea0553411f5c081b1fc41752115607bd0eefda7001fc7ce7e42967a99c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13754, "end_char_idx": 16529, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Category: Vegetables
```  

In the example above we instruct the model to classify the image into a single category, however, you can also prompt the model to do rank classification.

## Image-guided text generation

For more creative applications, you can use image-guided text generation to generate text based on an image. This can be 
useful to create descriptions of products, ads, descriptions of a scene, etc. 

Let's prompt IDEFICS to write a story based on a simple image of a red door: 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-story-generation.jpg" alt="Image of a red door with a pumpkin on the steps"/>
</div>

Photo by [Craig Tidball](https://unsplash.com/@devonshiremedia).
  
```py
>>> prompt = ["Instruction: Use the image to write a story. \n",
...     "https://images.unsplash.com/photo-1517086822157-2b0358e7684a?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2203&q=80",
...     "Story: \n"]

>>> inputs = processor(prompt, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, num_beams=2, max_new_tokens=200, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> print(generated_text[0]) 
Instruction: Use the image to write a story. 
 Story: 
Once upon a time, there was a little girl who lived in a house with a red door.  She loved her red door.  It was the prettiest door in the whole world.

One day, the little girl was playing in her yard when she noticed a man standing on her doorstep.  He was wearing a long black coat and a top hat.

The little girl ran inside and told her mother about the man.

Her mother said, “Don’t worry, honey.  He’s just a friendly ghost.”

The little girl wasn’t sure if she believed her mother, but she went outside anyway.

When she got to the door, the man was gone.

The next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep.

He was wearing a long black coat and a top hat.

The little girl ran
```

Looks like IDEFICS noticed the pumpkin on the doorstep and went with a spooky Halloween story about a ghost.

<Tip>

For longer outputs like this, you will greatly benefit from tweaking the text generation strategy. This can help 
you significantly improve the quality of the generated output. Check out [Text generation strategies](../generation_strategies) 
to learn more. 
</Tip>

## Running inference in batch mode

All of the earlier sections illustrated IDEFICS for a single example.


key: _node_content
string_value:
{"id_": "36d6d607-3d26-4d35-9762-e16ebdab0351", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6fda5281-4693-4f63-9332-ec5f94eb56ad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "8ea45a9dd57cb1c0dc7350805d9fe4b41cf003e23f85455d49f30db6fc51861f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1a6afd39-6f53-46c6-b1c0-1d27f5d7c7fb", "node_type": "1", "metadata": {}, "hash": "3caf6a68132596513471071bf1e0996a13ea2fc60dab38e30eb2d8189643694c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15282, "end_char_idx": 17893, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Story: 
Once upon a time, there was a little girl who lived in a house with a red door.  She loved her red door.  It was the prettiest door in the whole world.

One day, the little girl was playing in her yard when she noticed a man standing on her doorstep.  He was wearing a long black coat and a top hat.

The little girl ran inside and told her mother about the man.

Her mother said, “Don’t worry, honey.  He’s just a friendly ghost.”

The little girl wasn’t sure if she believed her mother, but she went outside anyway.

When she got to the door, the man was gone.

The next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep.

He was wearing a long black coat and a top hat.

The little girl ran
```

Looks like IDEFICS noticed the pumpkin on the doorstep and went with a spooky Halloween story about a ghost.

<Tip>

For longer outputs like this, you will greatly benefit from tweaking the text generation strategy. This can help 
you significantly improve the quality of the generated output. Check out [Text generation strategies](../generation_strategies) 
to learn more. 
</Tip>

## Running inference in batch mode

All of the earlier sections illustrated IDEFICS for a single example. In a very similar fashion, you can run inference 
for a batch of examples by passing a list of prompts:

```py
>>> prompts = [
...     [   "https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3501&q=80",
...         "This is an image of ",
...     ],
...     [   "https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",
...         "This is an image of ",
...     ],
...     [   "https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=3540&q=80",
...         "This is an image of ",
...     ],
... ]

>>> inputs = processor(prompts, return_tensors="pt").to("cuda")
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, max_new_tokens=10, bad_words_ids=bad_words_ids)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> for i,t in enumerate(generated_text):
...     print(f"{i}:\n{t}\n") 
0:
This is an image of the Eiffel Tower in Paris, France.

1:
This is an image of a couple on a picnic blanket.


key: _node_content
string_value:
{"id_": "1a6afd39-6f53-46c6-b1c0-1d27f5d7c7fb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "36d6d607-3d26-4d35-9762-e16ebdab0351", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "c67678b28c19b6750e75084ff1a0c524c9cc087c53a18025d1d4555a08ae8cd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17841, "end_char_idx": 20184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1:
This is an image of a couple on a picnic blanket.

2:
This is an image of a vegetable stand.
```

## IDEFICS instruct for conversational use

For conversational use cases, you can find fine-tuned instructed versions of the model on the 🤗 Hub: 
`HuggingFaceM4/idefics-80b-instruct` and `HuggingFaceM4/idefics-9b-instruct`.

These checkpoints are the result of fine-tuning the respective base models on a mixture of supervised and instruction 
fine-tuning datasets, which boosts the downstream performance while making the models more usable in conversational settings.

The use and prompting for the conversational use is very similar to using the base models: 

```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor

>>> device = "cuda" if torch.cuda.is_available() else "cpu"

>>> checkpoint = "HuggingFaceM4/idefics-9b-instruct"
>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> prompts = [
...     [
...         "User: What is in this image?",
...         "https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG",
...         "<end_of_utterance>",

...         "\nAssistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground.<end_of_utterance>",

...         "\nUser:",
...         "https://static.wikia.nocookie.net/asterix/images/2/25/R22b.gif/revision/latest?cb=20110815073052",
...         "And who is that?<end_of_utterance>",

...         "\nAssistant:",
...     ],
... ]

>>> # --batched mode
>>> inputs = processor(prompts, add_end_of_utterance_token=False, return_tensors="pt").to(device)
>>> # --single sample mode
>>> # inputs = processor(prompts[0], return_tensors="pt").to(device)

>>> # Generation args
>>> exit_condition = processor.tokenizer("<end_of_utterance>", add_special_tokens=False).input_ids
>>> bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

>>> generated_ids = model.generate(**inputs, eos_token_id=exit_condition, bad_words_ids=bad_words_ids, max_length=100)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> for i, t in enumerate(generated_text):
...     print(f"{i}:\n{t}\n")
```


key: _node_content
string_value:
{"id_": "c3607953-6210-4713-977d-434be26c14f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25c28460-c9a2-417f-95bd-963db38738a4", "node_type": "1", "metadata": {}, "hash": "d69b9fbd8f133968faf60fad49c0cf076cb76dd8d11684fbd15c54d237a1639c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Summarization

[[open-in-colab]]

<Youtube id="yHnr5Dk2zCI"/>

Summarization creates a shorter version of a document or an article that captures all the important information. Along with translation, it is another example of a task that can be formulated as a sequence-to-sequence task. Summarization can be:

- Extractive: extract the most relevant information from a document.
- Abstractive: generate new text that captures the most relevant information.

This guide will show you how to:

1. Finetune [T5](https://huggingface.co/google-t5/t5-small) on the California state bill subset of the [BillSum](https://huggingface.co/datasets/billsum) dataset for abstractive summarization.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/summarization)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate rouge_score
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load BillSum dataset

Start by loading the smaller California state bill subset of the BillSum dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> billsum = load_dataset("billsum", split="ca_test")
```

Split the dataset into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> billsum = billsum.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> billsum["train"][0]
{'summary': 'Existing law authorizes state agencies to enter into contracts for the acquisition of goods or services upon approval by the Department of General Services. Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\nThis bill would provide that no reimbursement is required by this act for a specified reason.',
 'text': 'The people of the State of California do enact as follows:\n\n\nSECTION 1.\nSection 10295.35 is added to the Public Contract Code, to read:\n10295.35.\n(a) (1) Notwithstanding any other law,


key: _node_content
string_value:
{"id_": "25c28460-c9a2-417f-95bd-963db38738a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3607953-6210-4713-977d-434be26c14f8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c7158908fd4c06f7316c01779fa4d818caec8dbdd842f445f658e9a49991f34e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda", "node_type": "1", "metadata": {}, "hash": "c9ac8e7815f0574e35d09a6dd26f2828a1c4909300a12220195a21dade7bd7da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1919, "end_char_idx": 5868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\nThis bill would provide that no reimbursement is required by this act for a specified reason.',
 'text': 'The people of the State of California do enact as follows:\n\n\nSECTION 1.\nSection 10295.35 is added to the Public Contract Code, to read:\n10295.35.\n(a) (1) Notwithstanding any other law, a state agency shall not enter into any contract for the acquisition of goods or services in the amount of one hundred thousand dollars ($100,000) or more with a contractor that, in the provision of benefits, discriminates between employees on the basis of an employee’s or dependent’s actual or perceived gender identity, including, but not limited to, the employee’s or dependent’s identification as transgender.\n(2) For purposes of this section, “contract” includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\n(4) The requirements of this section shall apply only to those portions of a contractor’s operations that occur under any of the following conditions:\n(A) Within the state.\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor’s presence at that location is connected to a contract with the state.\n(C) Elsewhere in the United States where work related to a state contract is being performed.\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor’s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with,


key: _node_content
string_value:
{"id_": "f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25c28460-c9a2-417f-95bd-963db38738a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "92b25a146fdf7a84d567208b5a108d27dc64c3515bc360a477842cc7b18abf2f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "62a9bc2a-f3d8-46b2-b577-24c791606df5", "node_type": "1", "metadata": {}, "hash": "d06e8648aeacf19c052162a6d4ea3dd77baecd5b96e99a2427822bf7e0b9c80b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3855, "end_char_idx": 7575, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
but not limited to, the employee’s or dependent’s identification as transgender.\n(2) For purposes of this section, “contract” includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\n(4) The requirements of this section shall apply only to those portions of a contractor’s operations that occur under any of the following conditions:\n(A) Within the state.\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor’s presence at that location is connected to a contract with the state.\n(C) Elsewhere in the United States where work related to a state contract is being performed.\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor’s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.


key: _node_content
string_value:
{"id_": "62a9bc2a-f3d8-46b2-b577-24c791606df5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4dc6a5e-7db8-46dc-bcf8-165cc2dbeeda", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "915f288c389674783d90f69bae2a86f2b199aadf4cb9a2869cf4103229d01776", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "108e24d3-3364-42a2-b344-b54672ff6863", "node_type": "1", "metadata": {}, "hash": "e184f27f91ed33251037adbf26dbdde4ba8d1122c0f70d760ea4c9dc986bdfb1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5541, "end_char_idx": 9360, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\nSEC. 2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}


key: _node_content
string_value:
{"id_": "108e24d3-3364-42a2-b344-b54672ff6863", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "62a9bc2a-f3d8-46b2-b577-24c791606df5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "cc5df8dfdb3f11e8041b33b70a4d4429c81cf1fc345eea286c0d315a6c41b24b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a79ab83f-ba93-4d89-ac75-06fe51fe7cee", "node_type": "1", "metadata": {}, "hash": "bdd516d048cb095cdf53b9aa952df7a6cbe211995fb48bd543965af445200623", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7442, "end_char_idx": 10203, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\nSEC. 2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}
```

There are two fields that you'll want to use:

- `text`: the text of the bill which'll be the input to the model.
- `summary`: a condensed version of `text` which'll be the model target.

## Preprocess

The next step is to load a T5 tokenizer to process `text` and `summary`:

```py
>>> from transformers import AutoTokenizer

>>> checkpoint = "google-t5/t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
```

The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a summarization task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Use the keyword `text_target` argument when tokenizing labels.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
>>> prefix = "summarize: "


key: _node_content
string_value:
{"id_": "a79ab83f-ba93-4d89-ac75-06fe51fe7cee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "108e24d3-3364-42a2-b344-b54672ff6863", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "dd0dda660f36abd526a9a23cfe6282da7d88765cd50f136883e97c27ed26c46d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6ae5f4c4-7b08-4996-be0c-7b354a61deb8", "node_type": "1", "metadata": {}, "hash": "c67ea74fee14e2d3caffe00d7a6ea3311b1f5f8c114cbef7d0fef2f8de4a0fe6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8510, "end_char_idx": 12162, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}
```

There are two fields that you'll want to use:

- `text`: the text of the bill which'll be the input to the model.
- `summary`: a condensed version of `text` which'll be the model target.

## Preprocess

The next step is to load a T5 tokenizer to process `text` and `summary`:

```py
>>> from transformers import AutoTokenizer

>>> checkpoint = "google-t5/t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
```

The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a summarization task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Use the keyword `text_target` argument when tokenizing labels.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
>>> prefix = "summarize: "


>>> def preprocess_function(examples):
...     inputs = [prefix + doc for doc in examples["text"]]
...     model_inputs = tokenizer(inputs, max_length=1024, truncation=True)

...     labels = tokenizer(text_target=examples["summary"], max_length=128, truncation=True)

...     model_inputs["labels"] = labels["input_ids"]
...     return model_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_billsum = billsum.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorForSeq2Seq`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>

```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)
```
</pt>
<tf>

```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [ROUGE](https://huggingface.co/spaces/evaluate-metric/rouge) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> rouge = evaluate.load("rouge")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the ROUGE metric:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "6ae5f4c4-7b08-4996-be0c-7b354a61deb8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a79ab83f-ba93-4d89-ac75-06fe51fe7cee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "a9bcc77ad663fe4b59591c2ec2f105bfa391d78dd501d87a6fc882956688feae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b798ecf-ee45-4623-83ec-d0a104df1619", "node_type": "1", "metadata": {}, "hash": "1fc095916afa675f55c71c898912f5e8a12a05820a51c9f84f0ce208ff8be328", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12165, "end_char_idx": 15070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

...     result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)

...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]
...     result["gen_len"] = np.mean(prediction_lens)

...     return {k: round(v, 4) for k, v in result.items()}
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:

```py
>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the ROUGE metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_billsum_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=4,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_billsum["train"],
...     eval_dataset=tokenized_billsum["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "7b798ecf-ee45-4623-83ec-d0a104df1619", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6ae5f4c4-7b08-4996-be0c-7b354a61deb8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfbbe55fa36f9f4290779c9e7e000f9cd289982a015cbcd362139e0c9df294a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "682ae8b1-c9fc-4879-ab43-2c537c8894db", "node_type": "1", "metadata": {}, "hash": "76f7488be7188048a3312e3b809ec8bb3c8e382229592e80113100cfe3b302de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13570, "end_char_idx": 16590, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the ROUGE metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_billsum_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=4,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_billsum["train"],
...     eval_dataset=tokenized_billsum["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_billsum["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_billsum["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the ROUGE score from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).


key: _node_content
string_value:
{"id_": "682ae8b1-c9fc-4879-ab43-2c537c8894db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b798ecf-ee45-4623-83ec-d0a104df1619", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "9e018253aeb8aa9f16bce63257b76d4b5819840f2864aa3e4a37b1a7ca8198e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c3865de-2a35-4b27-a657-b422679485cb", "node_type": "1", "metadata": {}, "hash": "063f3f076384dff4a8b28e20d5c36e2d8758bfd57f5946327c7d3786d17d860d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15072, "end_char_idx": 18303, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_billsum["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_billsum["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the ROUGE score from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_billsum_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for summarization, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on.


key: _node_content
string_value:
{"id_": "4c3865de-2a35-4b27-a657-b422679485cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "682ae8b1-c9fc-4879-ab43-2c537c8894db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "74a71bb1dedafba0f2f7cc50a5826c97194ab0fac0e06be36528742aaf47aa71", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "423b4164-731a-48da-9bec-979f3e4e010f", "node_type": "1", "metadata": {}, "hash": "16dcb01e3c780f50e688b261f9a493f76dbd2d759ed0de3a5afbbca6c2c676fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17302, "end_char_idx": 19550, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for summarization, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on. For summarization you should prefix your input as shown below:

```py
>>> text = "summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for summarization with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> summarizer = pipeline("summarization", model="stevhliu/my_awesome_billsum_model")
>>> summarizer(text)
[{"summary_text": "The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country."}]
```

You can also manually replicate the results of the `pipeline` if you'd like:


key: _node_content
string_value:
{"id_": "423b4164-731a-48da-9bec-979f3e4e010f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b9ed6f8-100f-4853-af25-09cf928ce987", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "c3d648d8a928b2b8b0f3f62054d010ad49f16e95df5d6868f3d8266a0d71666e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4c3865de-2a35-4b27-a657-b422679485cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/summarization", "title": "Summarization", "tokens": 5008, "retrieve_doc": true, "source": "transformers"}, "hash": "914e98619a7325698aadb8469f32e55408e0f3f687eb31c221fde0bca0549fad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18082, "end_char_idx": 21890, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to summarize. For T5, you need to prefix your input depending on the task you're working on. For summarization you should prefix your input as shown below:

```py
>>> text = "summarize: The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country. It'll lower the deficit and ask the ultra-wealthy and corporations to pay their fair share. And no one making under $400,000 per year will pay a penny more in taxes."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for summarization with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> summarizer = pipeline("summarization", model="stevhliu/my_awesome_billsum_model")
>>> summarizer(text)
[{"summary_text": "The Inflation Reduction Act lowers prescription drug costs, health care costs, and energy costs. It's the most aggressive action on tackling the climate crisis in American history, which will lift up American workers and create good-paying, union jobs across the country."}]
```

You can also manually replicate the results of the `pipeline` if you'd like:


<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> inputs = tokenizer(text, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import AutoModelForSeq2SeqLM

>>> model = AutoModelForSeq2SeqLM.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=False)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'the inflation reduction act lowers prescription drug costs, health care costs, and energy costs. it's the most aggressive action on tackling the climate crisis in american history. it will ask the ultra-wealthy and corporations to pay their fair share.'
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> inputs = tokenizer(text, return_tensors="tf").input_ids
```

Use the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained("stevhliu/my_awesome_billsum_model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=False)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'the inflation reduction act lowers prescription drug costs, health care costs, and energy costs. it's the most aggressive action on tackling the climate crisis in american history. it will ask the ultra-wealthy and corporations to pay their fair share.'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "65891829-a1aa-4a50-bcc6-f9a3e35da8e6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a67a987a-bddb-4978-99cf-136202525203", "node_type": "1", "metadata": {}, "hash": "6ec150254f33a00b4c0197617151c016cd3de138c6dfe16f684383b42f7ecf07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Mask Generation

Mask generation is the task of generating semantically meaningful masks for an image. 
This task is very similar to [image segmentation](semantic_segmentation), but many differences exist. Image segmentation models are trained on labeled datasets and are limited to the classes they have seen during training; they return a set of masks and corresponding classes, given an image. 

Mask generation models are trained on large amounts of data and operate in two modes. 
- Prompting mode: In this mode, the model takes in an image and a prompt, where a prompt can be a 2D point location (XY coordinates) in the image within an object or a bounding box surrounding an object. In prompting mode, the model only returns the mask over the object 
that the prompt is pointing out. 
- Segment Everything mode: In segment everything, given an image, the model generates every mask in the image. To do so, a grid of points is generated and overlaid on the image for inference. 

Mask generation task is supported by [Segment Anything Model (SAM)](model_doc/sam). It's a powerful model that consists of a Vision Transformer-based image encoder, a prompt encoder, and a two-way transformer mask decoder. Images and prompts are encoded, and the decoder takes these embeddings and generates valid masks. 

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sam.png" alt="SAM Architecture"/>
</div>

SAM serves as a powerful foundation model for segmentation as it has large data coverage. It is trained on 
[SA-1B](https://ai.meta.com/datasets/segment-anything/), a dataset with 1 million images and 1.1 billion masks. 

In this guide, you will learn how to:
- Infer in segment everything mode with batching,
- Infer in point prompting mode,
- Infer in box prompting mode.

First, let's install `transformers`:

```bash
pip install -q transformers
```

## Mask Generation Pipeline

The easiest way to infer mask generation models is to use the `mask-generation` pipeline.

```python
>>> from transformers import pipeline

>>> checkpoint = "facebook/sam-vit-base"
>>> mask_generator = pipeline(model=checkpoint, task="mask-generation")
```

Let's see the image.

```python
from PIL import Image
import requests

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Example Image"/>
</div>

Let's segment everything. `points-per-batch` enables parallel inference of points in segment everything mode. This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.


key: _node_content
string_value:
{"id_": "a67a987a-bddb-4978-99cf-136202525203", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65891829-a1aa-4a50-bcc6-f9a3e35da8e6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "1d4c7c8d31efdaf6fbe969e46d7b92aceb94ca80b9b1f8fdd25159f8178c7eaf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a7c8ee15-04e4-47f1-9da6-28769de1f85d", "node_type": "1", "metadata": {}, "hash": "c099248979b7f82b2d3fc917922ca409f250c854bb01029223269d4b40c174c5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1311, "end_char_idx": 4262, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sam.png" alt="SAM Architecture"/>
</div>

SAM serves as a powerful foundation model for segmentation as it has large data coverage. It is trained on 
[SA-1B](https://ai.meta.com/datasets/segment-anything/), a dataset with 1 million images and 1.1 billion masks. 

In this guide, you will learn how to:
- Infer in segment everything mode with batching,
- Infer in point prompting mode,
- Infer in box prompting mode.

First, let's install `transformers`:

```bash
pip install -q transformers
```

## Mask Generation Pipeline

The easiest way to infer mask generation models is to use the `mask-generation` pipeline.

```python
>>> from transformers import pipeline

>>> checkpoint = "facebook/sam-vit-base"
>>> mask_generator = pipeline(model=checkpoint, task="mask-generation")
```

Let's see the image.

```python
from PIL import Image
import requests

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Example Image"/>
</div>

Let's segment everything. `points-per-batch` enables parallel inference of points in segment everything mode. This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.

```python
masks = mask_generator(image, points_per_batch=128, pred_iou_thresh=0.88)
```

The `masks` looks like the following:

```bash
{'masks': [array([[False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         ...,
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False]]),
  array([[False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         ...,
'scores': tensor([0.9972, 0.9917,
        ...,
}
```

We can visualize them like this:

```python
import matplotlib.pyplot as plt

plt.imshow(image, cmap='gray')

for i, mask in enumerate(masks["masks"]):
    plt.imshow(mask, cmap='viridis', alpha=0.1, vmin=0, vmax=1)

plt.axis('off')
plt.show()
```

Below is the original image in grayscale with colorful maps overlaid. Very impressive.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_segmented.png" alt="Visualized"/>
</div>


key: _node_content
string_value:
{"id_": "a7c8ee15-04e4-47f1-9da6-28769de1f85d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a67a987a-bddb-4978-99cf-136202525203", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "41fcdf816c00785d00457985ae6f1e79a4aeb18f36cbce7ee4df1c31342bdc17", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c3e66b8-503e-48dc-b360-3ca289e8dc59", "node_type": "1", "metadata": {}, "hash": "f0ea727a0aa4d79b68a673172a4fea6e31fea884671222f24da93b79c0fc4de1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2764, "end_char_idx": 6701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This enables faster inference, but consumes more memory. Moreover, SAM only enables batching over points and not the images. `pred_iou_thresh` is the IoU confidence threshold where only the masks above that certain threshold are returned.

```python
masks = mask_generator(image, points_per_batch=128, pred_iou_thresh=0.88)
```

The `masks` looks like the following:

```bash
{'masks': [array([[False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         [False, False, False, ...,  True,  True,  True],
         ...,
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False]]),
  array([[False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         [False, False, False, ..., False, False, False],
         ...,
'scores': tensor([0.9972, 0.9917,
        ...,
}
```

We can visualize them like this:

```python
import matplotlib.pyplot as plt

plt.imshow(image, cmap='gray')

for i, mask in enumerate(masks["masks"]):
    plt.imshow(mask, cmap='viridis', alpha=0.1, vmin=0, vmax=1)

plt.axis('off')
plt.show()
```

Below is the original image in grayscale with colorful maps overlaid. Very impressive.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_segmented.png" alt="Visualized"/>
</div>


## Model Inference

### Point Prompting

You can also use the model without the pipeline. To do so, initialize the model and
the processor.

```python
from transformers import SamModel, SamProcessor
import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = SamModel.from_pretrained("facebook/sam-vit-base").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-base")
```

To do point prompting, pass the input point to the processor, then take the processor output
and pass it to the model for inference. To post-process the model output, pass the outputs and
`original_sizes` and `reshaped_input_sizes` we take from the processor's initial output. We need to pass these 
since the processor resizes the image, and the output needs to be extrapolated.

```python
input_points = [[[2592, 1728]]] # point location of the bee

inputs = processor(image, input_points=input_points, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)
masks = processor.image_processor.post_process_masks(outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu())
```
We can visualize the three masks in the `masks` output.

```python
import matplotlib.pyplot as plt
import numpy as np

fig, axes = plt.subplots(1, 4, figsize=(15, 5))

axes[0].imshow(image)
axes[0].set_title('Original Image')
mask_list = [masks[0][0][0].numpy(), masks[0][0][1].numpy(), masks[0][0][2].numpy()]

for i, mask in enumerate(mask_list, start=1):
    overlayed_image = np.array(image).copy()

    overlayed_image[:,:,0] = np.where(mask == 1, 255, overlayed_image[:,:,0])
    overlayed_image[:,:,1] = np.where(mask == 1, 0, overlayed_image[:,:,1])
    overlayed_image[:,:,2] = np.where(mask == 1, 0, overlayed_image[:,:,2])
    
    axes[i].imshow(overlayed_image)
    axes[i].set_title(f'Mask {i}')
for ax in axes:
    ax.axis('off')

plt.show()
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/masks.png" alt="Visualized"/>
</div>

### Box Prompting

You can also do box prompting in a similar fashion to point prompting. You can simply pass the input box in the format of a list
`[x_min, y_min, x_max, y_max]` format along with the image to the `processor`. Take the processor output and directly pass it 
to the model, then post-process the output again.


key: _node_content
string_value:
{"id_": "8c3e66b8-503e-48dc-b360-3ca289e8dc59", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d17f7d20-b435-4559-8f4c-b31a07de6ee9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "385f9b234ba753f61a1095d2d3cde2181f866dca68ccaf42a2dcf54ca0fb65b2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a7c8ee15-04e4-47f1-9da6-28769de1f85d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/mask_generation", "title": "Mask Generation", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "1496c6cfa0a677066af684ad48ac8af018ea5f1bfc479890e6b473a0edab61e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6704, "end_char_idx": 7995, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# bounding box around the bee
box = [2350, 1600, 2850, 2100]

inputs = processor(
        image,
        input_boxes=[[[box]]],
        return_tensors="pt"
    ).to("cuda")

with torch.no_grad():
    outputs = model(**inputs)

mask = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu()
)[0][0][0].numpy()
```

You can visualize the bounding box around the bee as shown below.

```python
import matplotlib.patches as patches

fig, ax = plt.subplots()
ax.imshow(image)

rectangle = patches.Rectangle((2350, 1600), 500, 500, linewidth=2, edgecolor='r', facecolor='none')
ax.add_patch(rectangle)
ax.axis("off")
plt.show()
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/bbox.png" alt="Visualized Bbox"/>
</div>

You can see the inference output below. 

```python
fig, ax = plt.subplots()
ax.imshow(image)
ax.imshow(mask, cmap='viridis', alpha=0.4)

ax.axis("off")
plt.show()
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/box_inference.png" alt="Visualized Inference"/>
</div>


key: _node_content
string_value:
{"id_": "f0e39e31-50eb-4daa-92aa-6cc6ff2b1bbe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "63f38317-e197-49f6-a801-741703eeaa50", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50745a6d-daa6-4802-aa98-33895d2194c4", "node_type": "1", "metadata": {}, "hash": "57f0f676a9f4b24e604e3b97d0e1f70e4861a8cb761b5e925fbca7da301c5826", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2898, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Zero-shot object detection

[[open-in-colab]]

Traditionally, models used for [object detection](object_detection) require labeled image datasets for training,
and are limited to detecting the set of classes from the training data.

Zero-shot object detection is supported by the [OWL-ViT](../model_doc/owlvit) model which uses a different approach. OWL-ViT
is an open-vocabulary object detector. It means that it can detect objects in images based on free-text queries without
the need to fine-tune the model on labeled datasets.

OWL-ViT leverages multi-modal representations to perform open-vocabulary detection. It combines [CLIP](../model_doc/clip) with
lightweight object classification and localization heads. Open-vocabulary detection is achieved by embedding free-text queries with the text encoder of CLIP and using them as input to the object classification and localization heads.
associate images and their corresponding textual descriptions, and ViT processes image patches as inputs. The authors
of OWL-ViT first trained CLIP from scratch and then fine-tuned OWL-ViT end to end on standard object detection datasets using
a bipartite matching loss.

With this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.

In this guide, you will learn how to use OWL-ViT:
- to detect objects based on text prompts
- for batch object detection
- for image-guided object detection

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q transformers
```

## Zero-shot object detection pipeline

The simplest way to try out inference with OWL-ViT is to use it in a [`pipeline`]. Instantiate a pipeline
for zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):

```python
>>> from transformers import pipeline

>>> checkpoint = "google/owlv2-base-patch16-ensemble"
>>> detector = pipeline(model=checkpoint, task="zero-shot-object-detection")
```

Next, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is
a part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset.

```py
>>> import skimage
>>> import numpy as np
>>> from PIL import Image

>>> image = skimage.data.astronaut()
>>> image = Image.fromarray(np.uint8(image)).convert("RGB")

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png" alt="Astronaut Eileen Collins"/>
</div>

Pass the image and the candidate object labels to look for to the pipeline.
Here we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for.


key: _node_content
string_value:
{"id_": "50745a6d-daa6-4802-aa98-33895d2194c4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "63f38317-e197-49f6-a801-741703eeaa50", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0e39e31-50eb-4daa-92aa-6cc6ff2b1bbe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "a96066c26dc7602891a516c6e79e22e57ee235d6cbf4bd552a5698c765c0bded", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "31ff1cf1-586b-4a08-8566-edc5d167368d", "node_type": "1", "metadata": {}, "hash": "0bedb9152c57476bafe862cc8c56f580398b1ab2552580308d8d30a181821169", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1167, "end_char_idx": 4636, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With this approach, the model can detect objects based on textual descriptions without prior training on labeled datasets.

In this guide, you will learn how to use OWL-ViT:
- to detect objects based on text prompts
- for batch object detection
- for image-guided object detection

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q transformers
```

## Zero-shot object detection pipeline

The simplest way to try out inference with OWL-ViT is to use it in a [`pipeline`]. Instantiate a pipeline
for zero-shot object detection from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit):

```python
>>> from transformers import pipeline

>>> checkpoint = "google/owlv2-base-patch16-ensemble"
>>> detector = pipeline(model=checkpoint, task="zero-shot-object-detection")
```

Next, choose an image you'd like to detect objects in. Here we'll use the image of astronaut Eileen Collins that is
a part of the [NASA](https://www.nasa.gov/multimedia/imagegallery/index.html) Great Images dataset.

```py
>>> import skimage
>>> import numpy as np
>>> from PIL import Image

>>> image = skimage.data.astronaut()
>>> image = Image.fromarray(np.uint8(image)).convert("RGB")

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_1.png" alt="Astronaut Eileen Collins"/>
</div>

Pass the image and the candidate object labels to look for to the pipeline.
Here we pass the image directly; other suitable options include a local path to an image or an image url. We also pass text descriptions for all items we want to query the image for. 

```py
>>> predictions = detector(
...     image,
...     candidate_labels=["human face", "rocket", "nasa badge", "star-spangled banner"],
... )
>>> predictions
[{'score': 0.3571370542049408,
  'label': 'human face',
  'box': {'xmin': 180, 'ymin': 71, 'xmax': 271, 'ymax': 178}},
 {'score': 0.28099656105041504,
  'label': 'nasa badge',
  'box': {'xmin': 129, 'ymin': 348, 'xmax': 206, 'ymax': 427}},
 {'score': 0.2110239565372467,
  'label': 'rocket',
  'box': {'xmin': 350, 'ymin': -1, 'xmax': 468, 'ymax': 288}},
 {'score': 0.13790413737297058,
  'label': 'star-spangled banner',
  'box': {'xmin': 1, 'ymin': 1, 'xmax': 105, 'ymax': 509}},
 {'score': 0.11950037628412247,
  'label': 'nasa badge',
  'box': {'xmin': 277, 'ymin': 338, 'xmax': 327, 'ymax': 380}},
 {'score': 0.10649408400058746,
  'label': 'rocket',
  'box': {'xmin': 358, 'ymin': 64, 'xmax': 424, 'ymax': 280}}]
```

Let's visualize the predictions:

```py
>>> from PIL import ImageDraw

>>> draw = ImageDraw.Draw(image)

>>> for prediction in predictions:
...     box = prediction["box"]
...     label = prediction["label"]
...     score = prediction["score"]

...     xmin, ymin, xmax, ymax = box.values()
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{label}: {round(score,2)}", fill="white")

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_2.png" alt="Visualized predictions on NASA image"/>
</div>

## Text-prompted zero-shot object detection by hand

Now that you've seen how to use the zero-shot object detection pipeline, let's replicate the same
result manually.


key: _node_content
string_value:
{"id_": "31ff1cf1-586b-4a08-8566-edc5d167368d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "63f38317-e197-49f6-a801-741703eeaa50", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50745a6d-daa6-4802-aa98-33895d2194c4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "59ba5ca7ff9b5f78b61c152e547936316f203ccbc49a52eea786b8c3333af528", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a91e4f9-b76c-4013-8b92-5418a6470afa", "node_type": "1", "metadata": {}, "hash": "f2679b3e1283b8389acede742e3aa00aca7adae05978ae60bf292065ec2d21f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4638, "end_char_idx": 7506, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?other=owlvit).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(checkpoint)
>>> processor = AutoProcessor.from_pretrained(checkpoint)
```

Let's take a different image to switch things up.

```py
>>> import requests

>>> url = "https://unsplash.com/photos/oj0zeY2Ltk4/download?ixid=MnwxMjA3fDB8MXxzZWFyY2h8MTR8fHBpY25pY3xlbnwwfHx8fDE2Nzc0OTE1NDk&force=true&w=640"
>>> im = Image.open(requests.get(url, stream=True).raw)
>>> im
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_3.png" alt="Beach photo"/>
</div>

Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the
image for the model by resizing and normalizing it, and a [`CLIPTokenizer`] that takes care of the text inputs.

```py
>>> text_queries = ["hat", "book", "sunglasses", "camera"]
>>> inputs = processor(text=text_queries, images=im, return_tensors="pt")
```

Pass the inputs through the model, post-process, and visualize the results. Since the image processor resized images before
feeding them to the model, you need to use the [`~OwlViTImageProcessor.post_process_object_detection`] method to make sure the predicted bounding
boxes have the correct coordinates relative to the original image:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = torch.tensor([im.size[::-1]])
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(im)

>>> scores = results["scores"].tolist()
>>> labels = results["labels"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[label]}: {round(score,2)}", fill="white")

>>> im
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Batch processing

You can pass multiple sets of images and text queries to search for different (or same) objects in several images.
Let's use both an astronaut image and the beach image together.
For batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,
PyTorch tensors, or NumPy arrays.


key: _node_content
string_value:
{"id_": "2a91e4f9-b76c-4013-8b92-5418a6470afa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "63f38317-e197-49f6-a801-741703eeaa50", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "31ff1cf1-586b-4a08-8566-edc5d167368d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "74ea051195fba5a8758d9926433f244481b93edc6ab1cf3aa5d61e59e9cea028", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15353950-3760-4509-aa89-781c39225fc3", "node_type": "1", "metadata": {}, "hash": "7ec40982ef3dc2111c501cdfbc6684f788849a2fc270c6be455be431fb0c863c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5996, "end_char_idx": 9215, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since the image processor resized images before
feeding them to the model, you need to use the [`~OwlViTImageProcessor.post_process_object_detection`] method to make sure the predicted bounding
boxes have the correct coordinates relative to the original image:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = torch.tensor([im.size[::-1]])
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(im)

>>> scores = results["scores"].tolist()
>>> labels = results["labels"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[label]}: {round(score,2)}", fill="white")

>>> im
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Batch processing

You can pass multiple sets of images and text queries to search for different (or same) objects in several images.
Let's use both an astronaut image and the beach image together.
For batch processing, you should pass text queries as a nested list to the processor and images as lists of PIL images,
PyTorch tensors, or NumPy arrays.

```py
>>> images = [image, im]
>>> text_queries = [
...     ["human face", "rocket", "nasa badge", "star-spangled banner"],
...     ["hat", "book", "sunglasses", "camera"],
... ]
>>> inputs = processor(text=text_queries, images=images, return_tensors="pt")
```

Previously for post-processing you passed the single image's size as a tensor, but you can also pass a tuple, or, in case
of several images, a list of tuples. Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`).

```py
>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = [x.size[::-1] for x in images]
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)

>>> image_idx = 1
>>> draw = ImageDraw.Draw(images[image_idx])

>>> scores = results[image_idx]["scores"].tolist()
>>> labels = results[image_idx]["labels"].tolist()
>>> boxes = results[image_idx]["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[image_idx][label]}: {round(score,2)}", fill="white")

>>> images[image_idx]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Image-guided object detection

In addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means
you can use an image query to find similar objects in the target image.


key: _node_content
string_value:
{"id_": "15353950-3760-4509-aa89-781c39225fc3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "63f38317-e197-49f6-a801-741703eeaa50", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a91e4f9-b76c-4013-8b92-5418a6470afa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "270f0c224ab48347ff571c1b6ad5e00539daa4382277a026662cc95b58b4d108", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a952c14-7a76-4301-9bbf-0ee8a74adf36", "node_type": "1", "metadata": {}, "hash": "579fbd9f59083f4ea1aafad225d23c489c474f135b633e9077cf401783306302", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7929, "end_char_idx": 10432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's create predictions for the two examples, and visualize the second one (`image_idx = 1`).

```py
>>> with torch.no_grad():
...     outputs = model(**inputs)
...     target_sizes = [x.size[::-1] for x in images]
...     results = processor.post_process_object_detection(outputs, threshold=0.1, target_sizes=target_sizes)

>>> image_idx = 1
>>> draw = ImageDraw.Draw(images[image_idx])

>>> scores = results[image_idx]["scores"].tolist()
>>> labels = results[image_idx]["labels"].tolist()
>>> boxes = results[image_idx]["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="red", width=1)
...     draw.text((xmin, ymin), f"{text_queries[image_idx][label]}: {round(score,2)}", fill="white")

>>> images[image_idx]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_4.png" alt="Beach photo with detected objects"/>
</div>

## Image-guided object detection

In addition to zero-shot object detection with text queries, OWL-ViT offers image-guided object detection. This means
you can use an image query to find similar objects in the target image.
Unlike text queries, only a single example image is allowed.

Let's take an image with two cats on a couch as a target image, and an image of a single cat
as a query:

```py
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image_target = Image.open(requests.get(url, stream=True).raw)

>>> query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
>>> query_image = Image.open(requests.get(query_url, stream=True).raw)
```

Let's take a quick look at the images:

```py
>>> import matplotlib.pyplot as plt

>>> fig, ax = plt.subplots(1, 2)
>>> ax[0].imshow(image_target)
>>> ax[1].imshow(query_image)
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png" alt="Cats"/>
</div>

In the preprocessing step, instead of text queries, you now need to use `query_images`:

```py
>>> inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")
```

For predictions, instead of passing the inputs to the model, pass them to [`~OwlViTForObjectDetection.image_guided_detection`]. Draw the predictions
as before except now there are no labels.


key: _node_content
string_value:
{"id_": "4a952c14-7a76-4301-9bbf-0ee8a74adf36", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "63f38317-e197-49f6-a801-741703eeaa50", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "7e8db6ca78f64dab105a404ac6bb403a05c857981260dd057dd944018eb8cee3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15353950-3760-4509-aa89-781c39225fc3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_object_detection", "title": "Zero-shot object detection", "tokens": 2993, "retrieve_doc": true, "source": "transformers"}, "hash": "2041335e14cf1710e017cb57d3a6b32fc465d0fb5a16145e7fc042e1f682fbdd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9133, "end_char_idx": 11224, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means
you can use an image query to find similar objects in the target image.
Unlike text queries, only a single example image is allowed.

Let's take an image with two cats on a couch as a target image, and an image of a single cat
as a query:

```py
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image_target = Image.open(requests.get(url, stream=True).raw)

>>> query_url = "http://images.cocodataset.org/val2017/000000524280.jpg"
>>> query_image = Image.open(requests.get(query_url, stream=True).raw)
```

Let's take a quick look at the images:

```py
>>> import matplotlib.pyplot as plt

>>> fig, ax = plt.subplots(1, 2)
>>> ax[0].imshow(image_target)
>>> ax[1].imshow(query_image)
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_5.png" alt="Cats"/>
</div>

In the preprocessing step, instead of text queries, you now need to use `query_images`:

```py
>>> inputs = processor(images=image_target, query_images=query_image, return_tensors="pt")
```

For predictions, instead of passing the inputs to the model, pass them to [`~OwlViTForObjectDetection.image_guided_detection`]. Draw the predictions
as before except now there are no labels.

```py
>>> with torch.no_grad():
...     outputs = model.image_guided_detection(**inputs)
...     target_sizes = torch.tensor([image_target.size[::-1]])
...     results = processor.post_process_image_guided_detection(outputs=outputs, target_sizes=target_sizes)[0]

>>> draw = ImageDraw.Draw(image_target)

>>> scores = results["scores"].tolist()
>>> boxes = results["boxes"].tolist()

>>> for box, score, label in zip(boxes, scores, labels):
...     xmin, ymin, xmax, ymax = box
...     draw.rectangle((xmin, ymin, xmax, ymax), outline="white", width=4)

>>> image_target
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/zero-sh-obj-detection_6.png" alt="Cats with bounding boxes"/>
</div>


key: _node_content
string_value:
{"id_": "045fa8fb-62f4-49bf-81e5-f6f6a30c0e6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "54c13277-7cb7-452d-973b-aa5b7468bde5", "node_type": "1", "metadata": {}, "hash": "6124ef1c5d9dbf649af3c2f030fd53cd1b84bcb0c53bccb420bccefdfb79a230", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2327, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image Segmentation

[[open-in-colab]]

<Youtube id="dKE8SIt9C-w"/>

Image segmentation models separate areas corresponding to different areas of interest in an image. These models work by assigning a label to each pixel. There are several types of segmentation: semantic segmentation, instance segmentation, and panoptic segmentation.

In this guide, we will:
1. [Take a look at different types of segmentation](#types-of-segmentation).
2. [Have an end-to-end fine-tuning example for semantic segmentation](#fine-tuning-a-model-for-segmentation).

Before you begin, make sure you have all the necessary libraries installed:

```py
# uncomment to install the necessary libraries
!pip install -q datasets transformers evaluate accelerate
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Types of Segmentation

Semantic segmentation assigns a label or class to every single pixel in an image. Let's take a look at a semantic segmentation model output. It will assign the same class to every instance of an object it comes across in an image, for example, all cats will be labeled as "cat" instead of "cat-1", "cat-2".
We can use transformers' image segmentation pipeline to quickly infer a semantic segmentation model. Let's take a look at the example image.

```python
from transformers import pipeline
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg" alt="Segmentation Input"/>
</div>

We will use [nvidia/segformer-b1-finetuned-cityscapes-1024-1024](https://huggingface.co/nvidia/segformer-b1-finetuned-cityscapes-1024-1024).

```python
semantic_segmentation = pipeline("image-segmentation", "nvidia/segformer-b1-finetuned-cityscapes-1024-1024")
results = semantic_segmentation(image)
results
```

The segmentation pipeline output includes a mask for every predicted class.


key: _node_content
string_value:
{"id_": "54c13277-7cb7-452d-973b-aa5b7468bde5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "045fa8fb-62f4-49bf-81e5-f6f6a30c0e6b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "5ba99c88e389ec3cde8471d8e4d55cc3cbe451f531d7ef37b11b4175f1e1b630", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6bb4781b-812c-40c2-900d-8d6df47ad19e", "node_type": "1", "metadata": {}, "hash": "332e480efc41cae58485573f76a0402fbc065de4b38e5b72408088e6d4f58cff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 858, "end_char_idx": 3383, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Types of Segmentation

Semantic segmentation assigns a label or class to every single pixel in an image. Let's take a look at a semantic segmentation model output. It will assign the same class to every instance of an object it comes across in an image, for example, all cats will be labeled as "cat" instead of "cat-1", "cat-2".
We can use transformers' image segmentation pipeline to quickly infer a semantic segmentation model. Let's take a look at the example image.

```python
from transformers import pipeline
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation_input.jpg" alt="Segmentation Input"/>
</div>

We will use [nvidia/segformer-b1-finetuned-cityscapes-1024-1024](https://huggingface.co/nvidia/segformer-b1-finetuned-cityscapes-1024-1024).

```python
semantic_segmentation = pipeline("image-segmentation", "nvidia/segformer-b1-finetuned-cityscapes-1024-1024")
results = semantic_segmentation(image)
results
```

The segmentation pipeline output includes a mask for every predicted class.
```bash
[{'score': None,
  'label': 'road',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sidewalk',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'building',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'wall',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'pole',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'traffic sign',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'vegetation',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'terrain',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```

Taking a look at the mask for the car class, we can see every car is classified with the same mask.


key: _node_content
string_value:
{"id_": "6bb4781b-812c-40c2-900d-8d6df47ad19e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54c13277-7cb7-452d-973b-aa5b7468bde5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "e8f03516ae1599e5d062fef92a3b681da7cf8e67f1c8ecda8a046f45860c268c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7503ae5-9844-4df0-899e-63f9a9eb1761", "node_type": "1", "metadata": {}, "hash": "d19353615ce3434092b174583dda05256f54f985c7c13b5ac4fbb23fc4319713", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2328, "end_char_idx": 4836, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
[{'score': None,
  'label': 'road',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sidewalk',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'building',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'wall',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'pole',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'traffic sign',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'vegetation',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'terrain',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': None,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```

Taking a look at the mask for the car class, we can see every car is classified with the same mask.

```python
results[-1]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/semantic_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

In instance segmentation, the goal is not to classify every pixel, but to predict a mask for **every instance of an object** in a given image. It works very similar to object detection, where there is a bounding box for every instance, there's a segmentation mask instead. We will use [facebook/mask2former-swin-large-cityscapes-instance](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-instance) for this.

```python
instance_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-instance")
results = instance_segmentation(image)
results
```

As you can see below, there are multiple cars classified, and there's no classification for pixels other than pixels that belong to car and person instances.

```bash
[{'score': 0.999944,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999945,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999652,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.903529,
  'label': 'person',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```
Checking out one of the car masks below.


key: _node_content
string_value:
{"id_": "f7503ae5-9844-4df0-899e-63f9a9eb1761", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6bb4781b-812c-40c2-900d-8d6df47ad19e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "1f825b0efe8e261f6f3bde900d9d62ebdec615d5fa4be2cb43f08d9b1750307a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4b3efff-2542-4b78-ada3-5df37495ef37", "node_type": "1", "metadata": {}, "hash": "e48832e7c3c106d2f31feb8510cce7b1d819460b8dfeda004749a8a783d24f53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3385, "end_char_idx": 5765, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results[-1]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/semantic_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

In instance segmentation, the goal is not to classify every pixel, but to predict a mask for **every instance of an object** in a given image. It works very similar to object detection, where there is a bounding box for every instance, there's a segmentation mask instead. We will use [facebook/mask2former-swin-large-cityscapes-instance](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-instance) for this.

```python
instance_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-instance")
results = instance_segmentation(image)
results
```

As you can see below, there are multiple cars classified, and there's no classification for pixels other than pixels that belong to car and person instances.

```bash
[{'score': 0.999944,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999945,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999652,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.903529,
  'label': 'person',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```
Checking out one of the car masks below.

```python
results[2]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/instance_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

Panoptic segmentation combines semantic segmentation and instance segmentation, where every pixel is classified into a class and an instance of that class, and there are multiple masks for each instance of a class. We can use [facebook/mask2former-swin-large-cityscapes-panoptic](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-panoptic) for this.

```python
panoptic_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-panoptic")
results = panoptic_segmentation(image)
results
```
As you can see below, we have more classes. We will later illustrate to see that every pixel is classified into one of the classes.


key: _node_content
string_value:
{"id_": "e4b3efff-2542-4b78-ada3-5df37495ef37", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7503ae5-9844-4df0-899e-63f9a9eb1761", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "9781af54940e88f586b3d844232b5c8dfda380abc9d858caf034ed50da57bb99", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c12511ca-acbe-4cb0-8bda-7921462dd3d0", "node_type": "1", "metadata": {}, "hash": "27b255274b441150644516f516d93eea7eb1c96885cdeb37882b514bd088ccc1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4401, "end_char_idx": 7045, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
[{'score': 0.999944,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999945,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999652,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.903529,
  'label': 'person',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```
Checking out one of the car masks below.

```python
results[2]["mask"]
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/instance_segmentation_output.png" alt="Semantic Segmentation Output"/>
</div>

Panoptic segmentation combines semantic segmentation and instance segmentation, where every pixel is classified into a class and an instance of that class, and there are multiple masks for each instance of a class. We can use [facebook/mask2former-swin-large-cityscapes-panoptic](https://huggingface.co/facebook/mask2former-swin-large-cityscapes-panoptic) for this.

```python
panoptic_segmentation = pipeline("image-segmentation", "facebook/mask2former-swin-large-cityscapes-panoptic")
results = panoptic_segmentation(image)
results
```
As you can see below, we have more classes. We will later illustrate to see that every pixel is classified into one of the classes.

```bash
[{'score': 0.999981,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999958,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.99997,
  'label': 'vegetation',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999575,
  'label': 'pole',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999958,
  'label': 'building',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999634,
  'label': 'road',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.996092,
  'label': 'sidewalk',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.999221,
  'label': 'car',
  'mask': <PIL.Image.Image image mode=L size=612x415>},
 {'score': 0.99987,
  'label': 'sky',
  'mask': <PIL.Image.Image image mode=L size=612x415>}]
```

Let's have a side by side comparison for all types of segmentation.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation-comparison.png" alt="Segmentation Maps Compared"/>
</div>

Seeing all types of segmentation, let's have a deep dive on fine-tuning a model for semantic segmentation.


key: _node_content
string_value:
{"id_": "c12511ca-acbe-4cb0-8bda-7921462dd3d0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4b3efff-2542-4b78-ada3-5df37495ef37", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "43c8a8cd4ee7f004e637d86c2997a65ee5ed3d66279d994c9d554840111c7649", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44e5c046-1784-4fe3-8bda-cece1b68d30b", "node_type": "1", "metadata": {}, "hash": "889e3b3d6625a79644a3da7340abdbaa71aa3916a57d49cd34bb75056a3950e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6721, "end_char_idx": 9110, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/segmentation-comparison.png" alt="Segmentation Maps Compared"/>
</div>

Seeing all types of segmentation, let's have a deep dive on fine-tuning a model for semantic segmentation.

Common real-world applications of semantic segmentation include training self-driving cars to identify pedestrians and important traffic information, identifying cells and abnormalities in medical imagery, and monitoring environmental changes from satellite imagery.

## Fine-tuning a Model for Segmentation

We will now:

1. Finetune [SegFormer](https://huggingface.co/docs/transformers/main/en/model_doc/segformer#segformer) on the [SceneParse150](https://huggingface.co/datasets/scene_parse_150) dataset.
2. Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-segmentation)

</Tip>


### Load SceneParse150 dataset

Start by loading a smaller subset of the SceneParse150 dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> ds = ds.train_test_split(test_size=0.2)
>>> train_ds = ds["train"]
>>> test_ds = ds["test"]
```

Then take a look at an example:

```py
>>> train_ds[0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,
 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,
 'scene_category': 368}

# view the image
>>> train_ds[0]["image"]
```

- `image`: a PIL image of the scene.
- `annotation`: a PIL image of the segmentation map, which is also the model's target.
- `scene_category`: a category id that describes the image scene like "kitchen" or "office". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.

You'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later.


key: _node_content
string_value:
{"id_": "44e5c046-1784-4fe3-8bda-cece1b68d30b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c12511ca-acbe-4cb0-8bda-7921462dd3d0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "997df885267a99b92a30e5b060c6a9a7b95111e27c1d5a44784f8932016f6fc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c747df0-bcae-4263-9f19-9c6d3c09a8f9", "node_type": "1", "metadata": {}, "hash": "1f9d8ee40e2d2fbf13dd9983ec089b2be97e662ce07b844c9119aa24aba5ec27", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7600, "end_char_idx": 9916, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-segmentation)

</Tip>


### Load SceneParse150 dataset

Start by loading a smaller subset of the SceneParse150 dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> ds = ds.train_test_split(test_size=0.2)
>>> train_ds = ds["train"]
>>> test_ds = ds["test"]
```

Then take a look at an example:

```py
>>> train_ds[0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x683 at 0x7F9B0C201F90>,
 'annotation': <PIL.PngImagePlugin.PngImageFile image mode=L size=512x683 at 0x7F9B0C201DD0>,
 'scene_category': 368}

# view the image
>>> train_ds[0]["image"]
```

- `image`: a PIL image of the scene.
- `annotation`: a PIL image of the segmentation map, which is also the model's target.
- `scene_category`: a category id that describes the image scene like "kitchen" or "office". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.

You'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the `id2label` and `label2id` dictionaries:

```py
>>> import json
>>> from pathlib import Path
>>> from huggingface_hub import hf_hub_download

>>> repo_id = "huggingface/label-files"
>>> filename = "ade20k-id2label.json"
>>> id2label = json.loads(Path(hf_hub_download(repo_id, filename, repo_type="dataset")).read_text())
>>> id2label = {int(k): v for k, v in id2label.items()}
>>> label2id = {v: k for k, v in id2label.items()}
>>> num_labels = len(id2label)
```

#### Custom dataset

You could also create and use your own dataset if you prefer to train with the [run_semantic_segmentation.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py) script instead of a notebook instance.


key: _node_content
string_value:
{"id_": "6c747df0-bcae-4263-9f19-9c6d3c09a8f9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44e5c046-1784-4fe3-8bda-cece1b68d30b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "6b6290eec4cf3f4dcf021af475cdd0db68fafac2a081cea325d85bac3780327b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "969eadb6-7288-4ae8-87b6-e917446fc07d", "node_type": "1", "metadata": {}, "hash": "42d94ed0b47c326fd19a6ff0cea5f0ad6679e4b4386896470be65edb34dbec02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8710, "end_char_idx": 11901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `annotation`: a PIL image of the segmentation map, which is also the model's target.
- `scene_category`: a category id that describes the image scene like "kitchen" or "office". In this guide, you'll only need `image` and `annotation`, both of which are PIL images.

You'll also want to create a dictionary that maps a label id to a label class which will be useful when you set up the model later. Download the mappings from the Hub and create the `id2label` and `label2id` dictionaries:

```py
>>> import json
>>> from pathlib import Path
>>> from huggingface_hub import hf_hub_download

>>> repo_id = "huggingface/label-files"
>>> filename = "ade20k-id2label.json"
>>> id2label = json.loads(Path(hf_hub_download(repo_id, filename, repo_type="dataset")).read_text())
>>> id2label = {int(k): v for k, v in id2label.items()}
>>> label2id = {v: k for k, v in id2label.items()}
>>> num_labels = len(id2label)
```

#### Custom dataset

You could also create and use your own dataset if you prefer to train with the [run_semantic_segmentation.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py) script instead of a notebook instance. The script requires:

1. a [`~datasets.DatasetDict`] with two [`~datasets.Image`] columns, "image" and "label"

     ```py
     from datasets import Dataset, DatasetDict, Image

     image_paths_train = ["path/to/image_1.jpg/jpg", "path/to/image_2.jpg/jpg", ..., "path/to/image_n.jpg/jpg"]
     label_paths_train = ["path/to/annotation_1.png", "path/to/annotation_2.png", ..., "path/to/annotation_n.png"]

     image_paths_validation = [...]
     label_paths_validation = [...]

     def create_dataset(image_paths, label_paths):
         dataset = Dataset.from_dict({"image": sorted(image_paths),
                                     "label": sorted(label_paths)})
         dataset = dataset.cast_column("image", Image())
         dataset = dataset.cast_column("label", Image())
         return dataset

     # step 1: create Dataset objects
     train_dataset = create_dataset(image_paths_train, label_paths_train)
     validation_dataset = create_dataset(image_paths_validation, label_paths_validation)

     # step 2: create DatasetDict
     dataset = DatasetDict({
          "train": train_dataset,
          "validation": validation_dataset,
          }
     )

     # step 3: push to Hub (assumes you have ran the huggingface-cli login command in a terminal/notebook)
     dataset.push_to_hub("your-name/dataset-repo")

     # optionally, you can push to a private repo on the Hub
     # dataset.push_to_hub("name of repo on the hub", private=True)
     ```

2. an id2label dictionary mapping the class integers to their class names

     ```py
     import json
     # simple example
     id2label = {0: 'cat', 1: 'dog'}
     with open('id2label.json', 'w') as fp:
     json.dump(id2label, fp)
     ```

As an example, take a look at this [example dataset](https://huggingface.co/datasets/nielsr/ade20k-demo) which was created with the steps shown above.

### Preprocess

The next step is to load a SegFormer image processor to prepare the images and annotations for the model.


key: _node_content
string_value:
{"id_": "969eadb6-7288-4ae8-87b6-e917446fc07d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c747df0-bcae-4263-9f19-9c6d3c09a8f9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "aa7869a99e353b4c272ba282649499def73dfcb786cb358bcc096639f9bfc973", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3b8f1056-23d6-4b14-9469-f281df739699", "node_type": "1", "metadata": {}, "hash": "0c9d533a75316f71ec98e4ecc3c9aa81d70fea132a66dc3e87d91e1ed3be6646", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11780, "end_char_idx": 14827, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Preprocess

The next step is to load a SegFormer image processor to prepare the images and annotations for the model. Some datasets, like this one, use the zero-index as the background class. However, the background class isn't actually included in the 150 classes, so you'll need to set `do_reduce_labels=True` to subtract one from all the labels. The zero-index is replaced by `255` so it's ignored by SegFormer's loss function:

```py
>>> from transformers import AutoImageProcessor

>>> checkpoint = "nvidia/mit-b0"
>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint, do_reduce_labels=True)
```

<frameworkcontent>
<pt>

It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting. In this guide, you'll use the [`ColorJitter`](https://pytorch.org/vision/stable/generated/torchvision.transforms.ColorJitter.html) function from [torchvision](https://pytorch.org/vision/stable/index.html) to randomly change the color properties of an image, but you can also use any image library you like.

```py
>>> from torchvision.transforms import ColorJitter

>>> jitter = ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.1)
```

Now create two preprocessing functions to prepare the images and annotations for the model. These functions convert the images into `pixel_values` and annotations to `labels`. For the training set, `jitter` is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the `images`, and only crops the `labels` because no data augmentation is applied during testing.

```py
>>> def train_transforms(example_batch):
...     images = [jitter(x) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


>>> def val_transforms(example_batch):
...     images = [x for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
```

To apply the `jitter` over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.set_transform`] function. The transform is applied on the fly which is faster and consumes less disk space:

```py
>>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.
In this guide, you'll use [`tf.image`](https://www.tensorflow.org/api_docs/python/tf/image) to randomly change the color properties of an image, but you can also use any image
library you like.
Define two separate transformation functions:
- training data transformations that include image augmentation
- validation data transformations that only transpose the images, since computer vision models in 🤗 Transformers expect channels-first layout

```py
>>> import tensorflow as tf


key: _node_content
string_value:
{"id_": "3b8f1056-23d6-4b14-9469-f281df739699", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "969eadb6-7288-4ae8-87b6-e917446fc07d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "0d08270beb704ce04c29d4470ee78a61335bad2e1bfb8310d66a8a615c20a488", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5126e8a7-accd-477e-9b14-70f8add1840d", "node_type": "1", "metadata": {}, "hash": "32889705aca16e332ef49ef69a1e05e88bdfef77fdf4ac66a6fa3b27f58b8bb9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13087, "end_char_idx": 15955, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These functions convert the images into `pixel_values` and annotations to `labels`. For the training set, `jitter` is applied before providing the images to the image processor. For the test set, the image processor crops and normalizes the `images`, and only crops the `labels` because no data augmentation is applied during testing.

```py
>>> def train_transforms(example_batch):
...     images = [jitter(x) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


>>> def val_transforms(example_batch):
...     images = [x for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
```

To apply the `jitter` over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.set_transform`] function. The transform is applied on the fly which is faster and consumes less disk space:

```py
>>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
It is common to apply some data augmentations to an image dataset to make a model more robust against overfitting.
In this guide, you'll use [`tf.image`](https://www.tensorflow.org/api_docs/python/tf/image) to randomly change the color properties of an image, but you can also use any image
library you like.
Define two separate transformation functions:
- training data transformations that include image augmentation
- validation data transformations that only transpose the images, since computer vision models in 🤗 Transformers expect channels-first layout

```py
>>> import tensorflow as tf


>>> def aug_transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.image.random_brightness(image, 0.25)
...     image = tf.image.random_contrast(image, 0.5, 2.0)
...     image = tf.image.random_saturation(image, 0.75, 1.25)
...     image = tf.image.random_hue(image, 0.1)
...     image = tf.transpose(image, (2, 0, 1))
...     return image


>>> def transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.transpose(image, (2, 0, 1))
...     return image
```

Next, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply
the image transformations and use the earlier loaded `image_processor` to convert the images into `pixel_values` and
annotations to `labels`. `ImageProcessor` also takes care of resizing and normalizing the images.

```py
>>> def train_transforms(example_batch):
...     images = [aug_transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


key: _node_content
string_value:
{"id_": "5126e8a7-accd-477e-9b14-70f8add1840d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b8f1056-23d6-4b14-9469-f281df739699", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "24a96b9c6e47a6a34ce21abf93dc701d6e2e255c230f69f59bc6608b6e77b5c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "365fa7ba-97a3-4db0-92d7-050cb6ce1582", "node_type": "1", "metadata": {}, "hash": "f64919519d54ee46dcc789813149f5d5cf857cdbd88e3dde70f071cc4d69c7f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14830, "end_char_idx": 18339, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def aug_transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.image.random_brightness(image, 0.25)
...     image = tf.image.random_contrast(image, 0.5, 2.0)
...     image = tf.image.random_saturation(image, 0.75, 1.25)
...     image = tf.image.random_hue(image, 0.1)
...     image = tf.transpose(image, (2, 0, 1))
...     return image


>>> def transforms(image):
...     image = tf.keras.utils.img_to_array(image)
...     image = tf.transpose(image, (2, 0, 1))
...     return image
```

Next, create two preprocessing functions to prepare batches of images and annotations for the model. These functions apply
the image transformations and use the earlier loaded `image_processor` to convert the images into `pixel_values` and
annotations to `labels`. `ImageProcessor` also takes care of resizing and normalizing the images.

```py
>>> def train_transforms(example_batch):
...     images = [aug_transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs


>>> def val_transforms(example_batch):
...     images = [transforms(x.convert("RGB")) for x in example_batch["image"]]
...     labels = [x for x in example_batch["annotation"]]
...     inputs = image_processor(images, labels)
...     return inputs
```

To apply the preprocessing transformations over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.set_transform`] function.
The transform is applied on the fly which is faster and consumes less disk space:

```py
>>> train_ds.set_transform(train_transforms)
>>> test_ds.set_transform(val_transforms)
```
</tf>
</frameworkcontent>

### Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load an evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [mean Intersection over Union](https://huggingface.co/spaces/evaluate-metric/accuracy) (IoU) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> metric = evaluate.load("mean_iou")
```

Then create a function to [`~evaluate.EvaluationModule.compute`] the metrics. Your predictions need to be converted to
logits first, and then reshaped to match the size of the labels before you can call [`~evaluate.EvaluationModule.compute`]:

<frameworkcontent>
<pt>

```py
>>> import numpy as np
>>> import torch
>>> from torch import nn

>>> def compute_metrics(eval_pred):
...     with torch.no_grad():
...         logits, labels = eval_pred
...         logits_tensor = torch.from_numpy(logits)
...         logits_tensor = nn.functional.interpolate(
...             logits_tensor,
...             size=labels.shape[-2:],
...             mode="bilinear",
...             align_corners=False,
...         ).argmax(dim=1)

...         pred_labels = logits_tensor.detach().cpu().numpy()
...         metrics = metric.compute(
...             predictions=pred_labels,
...             references=labels,
...             num_labels=num_labels,
...             ignore_index=255,
...             reduce_labels=False,
...         )
...         for key, value in metrics.items():
...             if isinstance(value, np.ndarray):
...                 metrics[key] = value.tolist()
...         return metrics
```

</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "365fa7ba-97a3-4db0-92d7-050cb6ce1582", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5126e8a7-accd-477e-9b14-70f8add1840d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "33c991f84f091f412b0d91f7bb70a35f8ca8b77f7d1a2221faa1253eb1db12ff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "592e9db3-5761-4cda-9d06-735559b62a94", "node_type": "1", "metadata": {}, "hash": "ee45fd45fe330e33ea671d3741c48858a1725e958bad49742727131c00d3fac7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18342, "end_char_idx": 20884, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<tf>

```py
>>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     logits = tf.transpose(logits, perm=[0, 2, 3, 1])
...     logits_resized = tf.image.resize(
...         logits,
...         size=tf.shape(labels)[1:],
...         method="bilinear",
...     )

...     pred_labels = tf.argmax(logits_resized, axis=-1)
...     metrics = metric.compute(
...         predictions=pred_labels,
...         references=labels,
...         num_labels=num_labels,
...         ignore_index=-1,
...         reduce_labels=image_processor.do_reduce_labels,
...     )

...     per_category_accuracy = metrics.pop("per_category_accuracy").tolist()
...     per_category_iou = metrics.pop("per_category_iou").tolist()

...     metrics.update({f"accuracy_{id2label[i]}": v for i, v in enumerate(per_category_accuracy)})
...     metrics.update({f"iou_{id2label[i]}": v for i, v in enumerate(per_category_iou)})
...     return {"val_" + k: v for k, v in metrics.items()}
```

</tf>
</frameworkcontent>

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

### Train
<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#finetune-with-trainer)!

</Tip>

You're ready to start training your model now! Load SegFormer with [`AutoModelForSemanticSegmentation`], and pass the model the mapping between label ids and label classes:

```py
>>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer

>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because this'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the IoU metric and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "592e9db3-5761-4cda-9d06-735559b62a94", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "365fa7ba-97a3-4db0-92d7-050cb6ce1582", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "aab139918b1c21b544f60d4e39a5acc5b3ad675b1896b056747981c6fa69c115", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87d4fb6e-1ba3-43f3-a707-f4c6d3b98974", "node_type": "1", "metadata": {}, "hash": "99650b7cf77a92208f6a77c57a65980fa003762a9a1fe65c26224f0e021c9099", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19473, "end_char_idx": 22350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Train
<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#finetune-with-trainer)!

</Tip>

You're ready to start training your model now! Load SegFormer with [`AutoModelForSemanticSegmentation`], and pass the model the mapping between label ids and label classes:

```py
>>> from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer

>>> model = AutoModelForSemanticSegmentation.from_pretrained(checkpoint, id2label=id2label, label2id=label2id)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because this'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the IoU metric and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="segformer-b0-scene-parse-150",
...     learning_rate=6e-5,
...     num_train_epochs=50,
...     per_device_train_batch_size=2,
...     per_device_eval_batch_size=2,
...     save_total_limit=3,
...     eval_strategy="steps",
...     save_strategy="steps",
...     save_steps=20,
...     eval_steps=20,
...     logging_steps=1,
...     eval_accumulation_steps=5,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=train_ds,
...     eval_dataset=test_ds,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pretrained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks to calculate metrics and upload your model to 🤗 Hub
6. Use the `fit()` method to run the training.


key: _node_content
string_value:
{"id_": "87d4fb6e-1ba3-43f3-a707-f4c6d3b98974", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "592e9db3-5761-4cda-9d06-735559b62a94", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "e8d9d759fea8cd60ef4148356d4bba788fe62b2a841f312e0f3f0d9fe1937586", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0327594-ce4d-40d9-88cd-0411c25f6d3d", "node_type": "1", "metadata": {}, "hash": "7029ad31362defd4ebe54fe57f2d3d563e0cab9abf5c0c5e242e680f5e19f722", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20834, "end_char_idx": 23366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="segformer-b0-scene-parse-150",
...     learning_rate=6e-5,
...     num_train_epochs=50,
...     per_device_train_batch_size=2,
...     per_device_eval_batch_size=2,
...     save_total_limit=3,
...     eval_strategy="steps",
...     save_strategy="steps",
...     save_steps=20,
...     eval_steps=20,
...     logging_steps=1,
...     eval_accumulation_steps=5,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=train_ds,
...     eval_dataset=test_ds,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pretrained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks to calculate metrics and upload your model to 🤗 Hub
6. Use the `fit()` method to run the training.

Start by defining the hyperparameters, optimizer and learning rate schedule:

```py
>>> from transformers import create_optimizer

>>> batch_size = 2
>>> num_epochs = 50
>>> num_train_steps = len(train_ds) * num_epochs
>>> learning_rate = 6e-5
>>> weight_decay_rate = 0.01

>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... )
```

Then, load SegFormer with [`TFAutoModelForSemanticSegmentation`] along with the label mappings, and compile it with the
optimizer. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
... )
>>> model.compile(optimizer=optimizer)  # No loss argument!


key: _node_content
string_value:
{"id_": "f0327594-ce4d-40d9-88cd-0411c25f6d3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87d4fb6e-1ba3-43f3-a707-f4c6d3b98974", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "65218c2ea38b83146cb36a8927bb2d78a8d5a1d2f373600b1943939059f8e5d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0e0a6f8-2d17-469c-95d0-8219f61660d5", "node_type": "1", "metadata": {}, "hash": "9a49d4c667aab8e8cff7116217554b6f2008b61206e7c039915cb2d739b64817", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21972, "end_char_idx": 25213, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pretrained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks to calculate metrics and upload your model to 🤗 Hub
6. Use the `fit()` method to run the training.

Start by defining the hyperparameters, optimizer and learning rate schedule:

```py
>>> from transformers import create_optimizer

>>> batch_size = 2
>>> num_epochs = 50
>>> num_train_steps = len(train_ds) * num_epochs
>>> learning_rate = 6e-5
>>> weight_decay_rate = 0.01

>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... )
```

Then, load SegFormer with [`TFAutoModelForSemanticSegmentation`] along with the label mappings, and compile it with the
optimizer. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
... )
>>> model.compile(optimizer=optimizer)  # No loss argument!
```

Convert your datasets to the `tf.data.Dataset` format using the [`~datasets.Dataset.to_tf_dataset`] and the [`DefaultDataCollator`]:

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator(return_tensors="tf")

>>> tf_train_dataset = train_ds.to_tf_dataset(
...     columns=["pixel_values", "label"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )

>>> tf_eval_dataset = test_ds.to_tf_dataset(
...     columns=["pixel_values", "label"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
```

To compute the accuracy from the predictions and push your model to the 🤗 Hub, use [Keras callbacks](../main_classes/keras_callbacks).
Pass your `compute_metrics` function to [`KerasMetricCallback`],
and use the [`PushToHubCallback`] to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )

>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)

>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... )
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>

### Inference

Great, now that you've finetuned a model, you can use it for inference!


key: _node_content
string_value:
{"id_": "e0e0a6f8-2d17-469c-95d0-8219f61660d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0327594-ce4d-40d9-88cd-0411c25f6d3d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "fe5eefbf1a32d7c8903d578a103ed34e84bb98770f76cd1ff9d5d6f4af66f6d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bda047c9-652b-4393-8a19-1a625d7acb2f", "node_type": "1", "metadata": {}, "hash": "7b6c332f760af9acedcc19b651959283d44cfbfdb2f81cdbc404410fce0a7921", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24129, "end_char_idx": 25749, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass your `compute_metrics` function to [`KerasMetricCallback`],
and use the [`PushToHubCallback`] to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )

>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)

>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... )
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>

### Inference

Great, now that you've finetuned a model, you can use it for inference!

Reload the dataset and load an image for inference.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
>>> ds = ds.train_test_split(test_size=0.2)
>>> test_ds = ds["test"]
>>> image = ds["test"][0]["image"]
>>> image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png" alt="Image of bedroom"/>
</div>

<frameworkcontent>
<pt>

We will now see how to infer without a pipeline.


key: _node_content
string_value:
{"id_": "bda047c9-652b-4393-8a19-1a625d7acb2f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0e0a6f8-2d17-469c-95d0-8219f61660d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "c5ad54cd33f49a77c0a9c777d832b8d5cc1519c815b4031a48d750a2c273684a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c99cb049-7b42-4a40-bb1a-d6f981b7dac5", "node_type": "1", "metadata": {}, "hash": "654a3cc059ce368316d40b0afbe0907eb3c10a56d33a23321eaed3c70be3c3ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24129, "end_char_idx": 27390, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass your `compute_metrics` function to [`KerasMetricCallback`],
and use the [`PushToHubCallback`] to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )

>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)

>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... )
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>

### Inference

Great, now that you've finetuned a model, you can use it for inference!

Reload the dataset and load an image for inference.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
>>> ds = ds.train_test_split(test_size=0.2)
>>> test_ds = ds["test"]
>>> image = ds["test"][0]["image"]
>>> image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png" alt="Image of bedroom"/>
</div>

<frameworkcontent>
<pt>

We will now see how to infer without a pipeline. Process the image with an image processor and place the `pixel_values` on a GPU:

```py
>>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  # use GPU if available, otherwise use a CPU
>>> encoding = image_processor(image, return_tensors="pt")
>>> pixel_values = encoding.pixel_values.to(device)
```

Pass your input to the model and return the `logits`:

```py
>>> outputs = model(pixel_values=pixel_values)
>>> logits = outputs.logits.cpu()
```

Next, rescale the logits to the original image size:

```py
>>> upsampled_logits = nn.functional.interpolate(
...     logits,
...     size=image.size[::-1],
...     mode="bilinear",
...     align_corners=False,
... )

>>> pred_seg = upsampled_logits.argmax(dim=1)[0]
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
Load an image processor to preprocess the image and return the input as TensorFlow tensors:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/scene_segmentation")
>>> inputs = image_processor(image, return_tensors="tf")
```

Pass your input to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("MariaK/scene_segmentation")
>>> logits = model(**inputs).logits
```

Next, rescale the logits to the original image size and apply argmax on the class dimension:
```py
>>> logits = tf.transpose(logits, [0, 2, 3, 1])

>>> upsampled_logits = tf.image.resize(
...     logits,
...     # We reverse the shape of `image` because `image.size` returns width and height.


key: _node_content
string_value:
{"id_": "c99cb049-7b42-4a40-bb1a-d6f981b7dac5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bda047c9-652b-4393-8a19-1a625d7acb2f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "7081e74c336b02ae379a944be528f76fd07cad6ca41171517c8a11ba526095e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "20b55705-bff3-47f4-8e53-5310b10cbed1", "node_type": "1", "metadata": {}, "hash": "c76b62c8021d262578030a4d6397245ab30aeafd02a7d38df60b9c3d9e7f4d1c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25750, "end_char_idx": 28315, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Process the image with an image processor and place the `pixel_values` on a GPU:

```py
>>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  # use GPU if available, otherwise use a CPU
>>> encoding = image_processor(image, return_tensors="pt")
>>> pixel_values = encoding.pixel_values.to(device)
```

Pass your input to the model and return the `logits`:

```py
>>> outputs = model(pixel_values=pixel_values)
>>> logits = outputs.logits.cpu()
```

Next, rescale the logits to the original image size:

```py
>>> upsampled_logits = nn.functional.interpolate(
...     logits,
...     size=image.size[::-1],
...     mode="bilinear",
...     align_corners=False,
... )

>>> pred_seg = upsampled_logits.argmax(dim=1)[0]
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
Load an image processor to preprocess the image and return the input as TensorFlow tensors:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/scene_segmentation")
>>> inputs = image_processor(image, return_tensors="tf")
```

Pass your input to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("MariaK/scene_segmentation")
>>> logits = model(**inputs).logits
```

Next, rescale the logits to the original image size and apply argmax on the class dimension:
```py
>>> logits = tf.transpose(logits, [0, 2, 3, 1])

>>> upsampled_logits = tf.image.resize(
...     logits,
...     # We reverse the shape of `image` because `image.size` returns width and height.
...     image.size[::-1],
... )

>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]
```

</tf>
</frameworkcontent>

To visualize the results, load the [dataset color palette](https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51) as `ade_palette()` that maps each class to their RGB values.

```py
def ade_palette():
  return np.asarray([
      [0, 0, 0],
      [120, 120, 120],
      [180, 120, 120],
      [6, 230, 230],
      [80, 50, 50],
      [4, 200, 3],
      [120, 120, 80],
      [140, 140, 140],
      [204, 5, 255],
      [230, 230, 230],
      [4, 250, 7],
      [224, 5, 255],
      [235, 255, 7],
      [150, 5, 61],
      [120, 120, 70],
      [8, 255, 51],
      [255, 6, 82],
      [143, 255, 140],
      [204, 255, 4],
      [255, 51, 7],
      [204, 70, 3],
      [0, 102, 200],
      [61, 230, 250],
      [255, 6,


key: _node_content
string_value:
{"id_": "20b55705-bff3-47f4-8e53-5310b10cbed1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c99cb049-7b42-4a40-bb1a-d6f981b7dac5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "b210651afc2275c70c55697fe703c97d5ca985c691b2515dcf5b30ecad6d9245", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2623d96-a2fa-47f0-b5ad-5d02885ee04b", "node_type": "1", "metadata": {}, "hash": "8a2243df478a5ee3610ff7bb2c6dfe62393290c9fe20d73c944b8f4849da0583", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27391, "end_char_idx": 29063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     image.size[::-1],
... )

>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]
```

</tf>
</frameworkcontent>

To visualize the results, load the [dataset color palette](https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51) as `ade_palette()` that maps each class to their RGB values.

```py
def ade_palette():
  return np.asarray([
      [0, 0, 0],
      [120, 120, 120],
      [180, 120, 120],
      [6, 230, 230],
      [80, 50, 50],
      [4, 200, 3],
      [120, 120, 80],
      [140, 140, 140],
      [204, 5, 255],
      [230, 230, 230],
      [4, 250, 7],
      [224, 5, 255],
      [235, 255, 7],
      [150, 5, 61],
      [120, 120, 70],
      [8, 255, 51],
      [255, 6, 82],
      [143, 255, 140],
      [204, 255, 4],
      [255, 51, 7],
      [204, 70, 3],
      [0, 102, 200],
      [61, 230, 250],
      [255, 6, 51],
      [11, 102, 255],
      [255, 7, 71],
      [255, 9, 224],
      [9, 7, 230],
      [220, 220, 220],
      [255, 9, 92],
      [112, 9, 255],
      [8, 255, 214],
      [7, 255, 224],
      [255, 184, 6],
      [10, 255, 71],
      [255, 41, 10],
      [7, 255, 255],
      [224, 255, 8],
      [102, 8, 255],
      [255, 61, 6],
      [255, 194, 7],
      [255, 122, 8],
      [0, 255, 20],
      [255, 8, 41],
      [255, 5, 153],
      [6, 51, 255],
      [235, 12, 255],
      [160, 150, 20],
      [0, 163, 255],
      [140, 140, 140],
      [250, 10, 15],
      [20, 255, 0],
      [31, 255, 0],
      [255, 31, 0],
      [255, 224, 0],
      [153, 255, 0],
      [0, 0, 255],
      [255, 71, 0],
      [0, 235, 255],
      [0, 173,


key: _node_content
string_value:
{"id_": "c2623d96-a2fa-47f0-b5ad-5d02885ee04b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "20b55705-bff3-47f4-8e53-5310b10cbed1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "4d233c881ff7a7be0af0e1bfda61a66d05e8d1a5465a510ebb03d40931a169dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23032ed7-c8f4-49b8-85f8-95b3acd358c5", "node_type": "1", "metadata": {}, "hash": "65527d728b9d77eff1556d1fdd0bdec48b564fcdf2bf2e970072adae620c89b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28313, "end_char_idx": 29781, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
6, 51],
      [11, 102, 255],
      [255, 7, 71],
      [255, 9, 224],
      [9, 7, 230],
      [220, 220, 220],
      [255, 9, 92],
      [112, 9, 255],
      [8, 255, 214],
      [7, 255, 224],
      [255, 184, 6],
      [10, 255, 71],
      [255, 41, 10],
      [7, 255, 255],
      [224, 255, 8],
      [102, 8, 255],
      [255, 61, 6],
      [255, 194, 7],
      [255, 122, 8],
      [0, 255, 20],
      [255, 8, 41],
      [255, 5, 153],
      [6, 51, 255],
      [235, 12, 255],
      [160, 150, 20],
      [0, 163, 255],
      [140, 140, 140],
      [250, 10, 15],
      [20, 255, 0],
      [31, 255, 0],
      [255, 31, 0],
      [255, 224, 0],
      [153, 255, 0],
      [0, 0, 255],
      [255, 71, 0],
      [0, 235, 255],
      [0, 173, 255],
      [31, 0, 255],
      [11, 200, 200],
      [255, 82, 0],
      [0, 255, 245],
      [0, 61, 255],
      [0, 255, 112],
      [0, 255, 133],
      [255, 0, 0],
      [255, 163, 0],
      [255, 102, 0],
      [194, 255, 0],
      [0, 143, 255],
      [51, 255, 0],
      [0, 82, 255],
      [0, 255, 41],
      [0, 255, 173],
      [10, 0, 255],
      [173, 255, 0],
      [0, 255, 153],
      [255, 92, 0],
      [255, 0, 255],
      [255, 0, 245],
      [255, 0, 102],
      [255, 173, 0],
      [255, 0, 20],
      [255, 184, 184],
      [0, 31, 255],
      [0, 255, 61],
      [0, 71, 255],
      [255, 0, 204],
      [0, 255, 194],
      [0, 255, 82],
      [0, 10, 255],
      [0, 112, 255],
      [51,


key: _node_content
string_value:
{"id_": "23032ed7-c8f4-49b8-85f8-95b3acd358c5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2623d96-a2fa-47f0-b5ad-5d02885ee04b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "a7eea018a59e9df652e1d36c2338730de37257cdd20f842f82c2bea79a299056", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "46e74bac-9f70-4b12-ab7c-6df655878c5c", "node_type": "1", "metadata": {}, "hash": "a1e219b327f2f232ebb37965203f0aa2e13799f883996ce398724fc32606ed52", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29038, "end_char_idx": 30512, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
235, 255],
      [0, 173, 255],
      [31, 0, 255],
      [11, 200, 200],
      [255, 82, 0],
      [0, 255, 245],
      [0, 61, 255],
      [0, 255, 112],
      [0, 255, 133],
      [255, 0, 0],
      [255, 163, 0],
      [255, 102, 0],
      [194, 255, 0],
      [0, 143, 255],
      [51, 255, 0],
      [0, 82, 255],
      [0, 255, 41],
      [0, 255, 173],
      [10, 0, 255],
      [173, 255, 0],
      [0, 255, 153],
      [255, 92, 0],
      [255, 0, 255],
      [255, 0, 245],
      [255, 0, 102],
      [255, 173, 0],
      [255, 0, 20],
      [255, 184, 184],
      [0, 31, 255],
      [0, 255, 61],
      [0, 71, 255],
      [255, 0, 204],
      [0, 255, 194],
      [0, 255, 82],
      [0, 10, 255],
      [0, 112, 255],
      [51, 0, 255],
      [0, 194, 255],
      [0, 122, 255],
      [0, 255, 163],
      [255, 153, 0],
      [0, 255, 10],
      [255, 112, 0],
      [143, 255, 0],
      [82, 0, 255],
      [163, 255, 0],
      [255, 235, 0],
      [8, 184, 170],
      [133, 0, 255],
      [0, 255, 92],
      [184, 0, 255],
      [255, 0, 31],
      [0, 184, 255],
      [0, 214, 255],
      [255, 0, 112],
      [92, 255, 0],
      [0, 224, 255],
      [112, 224, 255],
      [70, 184, 160],
      [163, 0, 255],
      [153, 0, 255],
      [71, 255, 0],
      [255, 0, 163],
      [255, 204, 0],
      [255, 0, 143],
      [0, 255, 235],
      [133, 255, 0],
      [255, 0, 235],
      [245, 0, 255],
      [255, 0, 122],
      [255, 245, 0],
      [10,


key: _node_content
string_value:
{"id_": "46e74bac-9f70-4b12-ab7c-6df655878c5c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23032ed7-c8f4-49b8-85f8-95b3acd358c5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "419c9751cc3eea09f8fb17102abd2990196f60f772d5328fbcbfe85cf96742a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ab9ca066-8e54-4b9d-a067-70dbcccb842e", "node_type": "1", "metadata": {}, "hash": "8820c311ab122c4c017562b695f18cc3f2cf85f41d7f71e754c9f88f6b5c3036", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
112, 255],
      [51, 0, 255],
      [0, 194, 255],
      [0, 122, 255],
      [0, 255, 163],
      [255, 153, 0],
      [0, 255, 10],
      [255, 112, 0],
      [143, 255, 0],
      [82, 0, 255],
      [163, 255, 0],
      [255, 235, 0],
      [8, 184, 170],
      [133, 0, 255],
      [0, 255, 92],
      [184, 0, 255],
      [255, 0, 31],
      [0, 184, 255],
      [0, 214, 255],
      [255, 0, 112],
      [92, 255, 0],
      [0, 224, 255],
      [112, 224, 255],
      [70, 184, 160],
      [163, 0, 255],
      [153, 0, 255],
      [71, 255, 0],
      [255, 0, 163],
      [255, 204, 0],
      [255, 0, 143],
      [0, 255, 235],
      [133, 255, 0],
      [255, 0, 235],
      [245, 0, 255],
      [255, 0, 122],
      [255, 245, 0],
      [10, 190, 212],
      [214, 255, 0],
      [0, 204, 255],
      [20, 0, 255],
      [255, 255, 0],
      [0, 153, 255],
      [0, 41, 255],
      [0, 255, 204],
      [41, 0, 255],
      [41, 255, 0],
      [173, 0, 255],
      [0, 245, 255],
      [71, 0, 255],
      [122, 0, 255],
      [0, 255, 184],
      [0, 92, 255],
      [184, 255, 0],
      [0, 133, 255],
      [255, 214, 0],
      [25, 194, 194],
      [102, 255, 0],
      [92, 0, 255],
  ])
```

Then you can combine and plot your image and the predicted segmentation map:

```py
>>> import matplotlib.pyplot as plt
>>> import numpy as np

>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)
>>> palette = np.array(ade_palette())
>>> for label, color in enumerate(palette):
.     color_seg[pred_seg == label, :] = color
>>> color_seg = color_seg[. ::-1]  # convert to BGR

>>> img = np.array(image) * 0.5 + color_seg * 0.


key: _node_content
string_value:
{"id_": "ab9ca066-8e54-4b9d-a067-70dbcccb842e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c93072b-42b4-4c50-bac8-8f2f8d5f938a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "29eb63b4cdaebee7daa5ed72185c11d247fa7bd9fe25b32d4a9cc31bd3bfea2d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46e74bac-9f70-4b12-ab7c-6df655878c5c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "transformers"}, "hash": "9489197698d5a35cd49a44d8cf073705044b39c44004ca34f708e170daa19e87", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0, 122],
      [255, 245, 0],
      [10, 190, 212],
      [214, 255, 0],
      [0, 204, 255],
      [20, 0, 255],
      [255, 255, 0],
      [0, 153, 255],
      [0, 41, 255],
      [0, 255, 204],
      [41, 0, 255],
      [41, 255, 0],
      [173, 0, 255],
      [0, 245, 255],
      [71, 0, 255],
      [122, 0, 255],
      [0, 255, 184],
      [0, 92, 255],
      [184, 255, 0],
      [0, 133, 255],
      [255, 214, 0],
      [25, 194, 194],
      [102, 255, 0],
      [92, 0, 255],
  ])
```

Then you can combine and plot your image and the predicted segmentation map:

```py
>>> import matplotlib.pyplot as plt
>>> import numpy as np

>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)
>>> palette = np.array(ade_palette())
>>> for label, color in enumerate(palette):
.     color_seg[pred_seg == label, :] = color
>>> color_seg = color_seg[. ::-1]  # convert to BGR

>>> img = np.array(image) * 0.5 + color_seg * 0.5  # plot the image with the segmentation map
>>> img = img.astype(np.uint8)

>>> plt.figure(figsize=(15, 10))
>>> plt.imshow(img)
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-preds.png" alt="Image of bedroom overlaid with segmentation map"/>
</div>


key: _node_content
string_value:
{"id_": "ac30ffe2-b3d5-4792-901f-ba1f72bb80cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e", "node_type": "1", "metadata": {}, "hash": "c469c64c686b38b6063bb340fd071e6777e3cc9aa72c916bf687c08dc232085d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image captioning

[[open-in-colab]]

Image captioning is the task of predicting a caption for a given image. Common real world applications of it include
aiding visually impaired people that can help them navigate through different situations. Therefore, image captioning
helps to improve content accessibility for people by describing images to them.

This guide will show you how to:

* Fine-tune an image captioning model.
* Use the fine-tuned model for inference. 

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate -q
pip install jiwer -q
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:


```python
from huggingface_hub import notebook_login

notebook_login()
```

## Load the Pokémon BLIP captions dataset

Use the 🤗 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset
in PyTorch, you can follow [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GIT/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb). 


```python
from datasets import load_dataset

ds = load_dataset("lambdalabs/pokemon-blip-captions")
ds
```
```bash
DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 833
    })
})
```

The dataset has two features, `image` and `text`.

<Tip>

Many image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training. 

</Tip>

Split the dataset’s train split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:


```python
ds = ds["train"].train_test_split(test_size=0.1)
train_ds = ds["train"]
test_ds = ds["test"]
```

Let's visualize a couple of samples from the training set. 


```python
from textwrap import wrap
import matplotlib.pyplot as plt
import numpy as np


def plot_images(images, captions):
    plt.figure(figsize=(20, 20))
    for i in range(len(images)):
        ax = plt.subplot(1, len(images), i + 1)
        caption = captions[i]
        caption = "\n".join(wrap(caption, 12))
        plt.title(caption)
        plt.imshow(images[i])
        plt.axis("off")


key: _node_content
string_value:
{"id_": "f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac30ffe2-b3d5-4792-901f-ba1f72bb80cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "63084dee75d6f3ef67a51b598853a291a78897a734d023accc3622aab45067a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5359143f-0a61-4bcc-9158-354e7eead63c", "node_type": "1", "metadata": {}, "hash": "f49ccd7582a09ead0a203446ff9618ecaa9be430bd6807e6068076d0b6a013ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 789, "end_char_idx": 3953, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from huggingface_hub import notebook_login

notebook_login()
```

## Load the Pokémon BLIP captions dataset

Use the 🤗 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset
in PyTorch, you can follow [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GIT/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb). 


```python
from datasets import load_dataset

ds = load_dataset("lambdalabs/pokemon-blip-captions")
ds
```
```bash
DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 833
    })
})
```

The dataset has two features, `image` and `text`.

<Tip>

Many image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training. 

</Tip>

Split the dataset’s train split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:


```python
ds = ds["train"].train_test_split(test_size=0.1)
train_ds = ds["train"]
test_ds = ds["test"]
```

Let's visualize a couple of samples from the training set. 


```python
from textwrap import wrap
import matplotlib.pyplot as plt
import numpy as np


def plot_images(images, captions):
    plt.figure(figsize=(20, 20))
    for i in range(len(images)):
        ax = plt.subplot(1, len(images), i + 1)
        caption = captions[i]
        caption = "\n".join(wrap(caption, 12))
        plt.title(caption)
        plt.imshow(images[i])
        plt.axis("off")


sample_images_to_visualize = [np.array(train_ds[i]["image"]) for i in range(5)]
sample_captions = [train_ds[i]["text"] for i in range(5)]
plot_images(sample_images_to_visualize, sample_captions)
```
    
<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_training_images_image_cap.png" alt="Sample training images"/>
</div>

## Preprocess the dataset

Since the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.

To do so, load the processor class associated with the model you are about to fine-tune. 

```python
from transformers import AutoProcessor

checkpoint = "microsoft/git-base"
processor = AutoProcessor.from_pretrained(checkpoint)
```

The processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption. 

```python
def transforms(example_batch):
    images = [x for x in example_batch["image"]]
    captions = [x for x in example_batch["text"]]
    inputs = processor(images=images, text=captions, padding="max_length")
    inputs.update({"labels": inputs["input_ids"]})
    return inputs


train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
```

With the dataset ready, you can now set up the model for fine-tuning. 

## Load a base model

Load the ["microsoft/git-base"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.


key: _node_content
string_value:
{"id_": "5359143f-0a61-4bcc-9158-354e7eead63c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f77dd0eb-f1f0-4d37-94fd-de6ddf4f841e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "a449ca6da5e097c1f201cd14a1cc2d886e1eed3bfd8267933b816b49bfae5acc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "659b88a0-009c-45c8-b919-2405bf890fe1", "node_type": "1", "metadata": {}, "hash": "1f1680a6ba28c4f5e499e7141a37120323fd0a735223608b73bd21b35475c047", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2357, "end_char_idx": 5146, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
sample_images_to_visualize = [np.array(train_ds[i]["image"]) for i in range(5)]
sample_captions = [train_ds[i]["text"] for i in range(5)]
plot_images(sample_images_to_visualize, sample_captions)
```
    
<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_training_images_image_cap.png" alt="Sample training images"/>
</div>

## Preprocess the dataset

Since the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.

To do so, load the processor class associated with the model you are about to fine-tune. 

```python
from transformers import AutoProcessor

checkpoint = "microsoft/git-base"
processor = AutoProcessor.from_pretrained(checkpoint)
```

The processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption. 

```python
def transforms(example_batch):
    images = [x for x in example_batch["image"]]
    captions = [x for x in example_batch["text"]]
    inputs = processor(images=images, text=captions, padding="max_length")
    inputs.update({"labels": inputs["input_ids"]})
    return inputs


train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
```

With the dataset ready, you can now set up the model for fine-tuning. 

## Load a base model

Load the ["microsoft/git-base"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.


```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(checkpoint)
```

## Evaluate

Image captioning models are typically evaluated with the [Rouge Score](https://huggingface.co/spaces/evaluate-metric/rouge) or [Word Error Rate](https://huggingface.co/spaces/evaluate-metric/wer). For this guide, you will use the Word Error Rate (WER). 

We use the 🤗 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to [this guide](https://huggingface.co/spaces/evaluate-metric/wer). 


```python
from evaluate import load
import torch

wer = load("wer")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predicted = logits.argmax(-1)
    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)
    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)
    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)
    return {"wer_score": wer_score}
```

## Train!

Now, you are ready to start fine-tuning the model. You will use the 🤗 [`Trainer`] for this. 

First, define the training arguments using [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "659b88a0-009c-45c8-b919-2405bf890fe1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5359143f-0a61-4bcc-9158-354e7eead63c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "1da929197ddfb4dce336d1b008e4ce305de8d79028943830ef5ea9c780dd479f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec3a9f8c-a654-42ae-8393-0d402a1cb619", "node_type": "1", "metadata": {}, "hash": "d47d08787f0017b2e6b4f1b639a84997b6effc52688873af0686e1c9e9071aed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3580, "end_char_idx": 6411, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
```

With the dataset ready, you can now set up the model for fine-tuning. 

## Load a base model

Load the ["microsoft/git-base"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.


```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(checkpoint)
```

## Evaluate

Image captioning models are typically evaluated with the [Rouge Score](https://huggingface.co/spaces/evaluate-metric/rouge) or [Word Error Rate](https://huggingface.co/spaces/evaluate-metric/wer). For this guide, you will use the Word Error Rate (WER). 

We use the 🤗 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to [this guide](https://huggingface.co/spaces/evaluate-metric/wer). 


```python
from evaluate import load
import torch

wer = load("wer")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predicted = logits.argmax(-1)
    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)
    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)
    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)
    return {"wer_score": wer_score}
```

## Train!

Now, you are ready to start fine-tuning the model. You will use the 🤗 [`Trainer`] for this. 

First, define the training arguments using [`TrainingArguments`].


```python
from transformers import TrainingArguments, Trainer

model_name = checkpoint.split("/")[1]

training_args = TrainingArguments(
    output_dir=f"{model_name}-pokemon",
    learning_rate=5e-5,
    num_train_epochs=50,
    fp16=True,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    gradient_accumulation_steps=2,
    save_total_limit=3,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    logging_steps=50,
    remove_unused_columns=False,
    push_to_hub=True,
    label_names=["labels"],
    load_best_model_at_end=True,
)
```

Then pass them along with the datasets and the model to 🤗 Trainer. 

```python
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    compute_metrics=compute_metrics,
)
```

To start training, simply call [`~Trainer.train`] on the [`Trainer`] object.

```python 
trainer.train()
```

You should see the training loss drop smoothly as training progresses.

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method so everyone can use your model:


```python
trainer.push_to_hub()
```

## Inference

Take a sample image from `test_ds` to test the model.


key: _node_content
string_value:
{"id_": "ec3a9f8c-a654-42ae-8393-0d402a1cb619", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32a9f40d-0fe2-4e78-90a6-12c4c4bd5c22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "91c7f9bca67878ef01d86e644d523e7e994743923414f046881da32189729c0f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "659b88a0-009c-45c8-b919-2405bf890fe1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "transformers"}, "hash": "6d836cea26a90adaeb76467296216abbe5e6145c18c4aba064226f0d6000dbfb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5149, "end_char_idx": 7385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import TrainingArguments, Trainer

model_name = checkpoint.split("/")[1]

training_args = TrainingArguments(
    output_dir=f"{model_name}-pokemon",
    learning_rate=5e-5,
    num_train_epochs=50,
    fp16=True,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    gradient_accumulation_steps=2,
    save_total_limit=3,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    logging_steps=50,
    remove_unused_columns=False,
    push_to_hub=True,
    label_names=["labels"],
    load_best_model_at_end=True,
)
```

Then pass them along with the datasets and the model to 🤗 Trainer. 

```python
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    compute_metrics=compute_metrics,
)
```

To start training, simply call [`~Trainer.train`] on the [`Trainer`] object.

```python 
trainer.train()
```

You should see the training loss drop smoothly as training progresses.

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method so everyone can use your model:


```python
trainer.push_to_hub()
```

## Inference

Take a sample image from `test_ds` to test the model.


```python
from PIL import Image
import requests

url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/pokemon.png"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/test_image_image_cap.png" alt="Test image"/>
</div>
    
Prepare image for the model.

```python
device = "cuda" if torch.cuda.is_available() else "cpu"

inputs = processor(images=image, return_tensors="pt").to(device)
pixel_values = inputs.pixel_values
```

Call [`generate`] and decode the predictions. 

```python
generated_ids = model.generate(pixel_values=pixel_values, max_length=50)
generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_caption)
```
```bash
a drawing of a pink and blue pokemon
```

Looks like the fine-tuned model generated a pretty good caption!


key: _node_content
string_value:
{"id_": "f5b93891-1eb4-4782-b415-c8e9ea917cca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "hash": "e9c2f2f6611fc285f25ce1b46f84a11985fff7798a572e3349601e8720a6d5de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f87c4f15-e8c6-467a-9b82-bd4e1678299d", "node_type": "1", "metadata": {}, "hash": "dfff0139cabab5dd294e0f21af1aacc0a2872d281258f9ba9804f68f50435191", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2561, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image-text-to-text

[[open-in-colab]]

Image-text-to-text models, also known as vision language models (VLMs), are language models that take an image input. These models can tackle various tasks, from visual question answering to image segmentation. This task shares many similarities with image-to-text, but with some overlapping use cases like image captioning. Image-to-text models only take image inputs and often accomplish a specific task, whereas VLMs take open-ended text and image inputs and are more generalist models.

In this guide, we provide a brief overview of VLMs and show how to use them with Transformers for inference.

To begin with, there are multiple types of VLMs:
- base models used for fine-tuning
- chat fine-tuned models for conversation
- instruction fine-tuned models

This guide focuses on inference with an instruction-tuned model. 

Let's begin installing the dependencies.

```bash
pip install -q transformers accelerate flash_attn 
```

Let's initialize the model and the processor. 

```python
from transformers import AutoProcessor, Idefics2ForConditionalGeneration
import torch

device = torch.device("cuda")
model = Idefics2ForConditionalGeneration.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to(device)

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
```

This model has a [chat template](./chat_templating) that helps user parse chat outputs. Moreover, the model can also accept multiple images as input in a single conversation or message. We will now prepare the inputs. 

The image inputs look like the following.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png" alt="Two cats sitting on a net"/>
</div>

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="A bee on a pink flower"/>
</div>


```python
from PIL import Image
import requests

img_urls =["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
           "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"]
images = [Image.open(requests.get(img_urls[0], stream=True).raw),
          Image.open(requests.get(img_urls[1], stream=True).raw)]
```

Below is an example of the chat template. We can feed conversation turns and the last message as an input by appending it at the end of the template.


key: _node_content
string_value:
{"id_": "f87c4f15-e8c6-467a-9b82-bd4e1678299d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "hash": "e9c2f2f6611fc285f25ce1b46f84a11985fff7798a572e3349601e8720a6d5de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5b93891-1eb4-4782-b415-c8e9ea917cca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "hash": "afc3fac2f189afead04497a5d766e332c068412ac730efba1e48513a73a56145", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48bc4f69-dee7-495b-bcd5-565659126c20", "node_type": "1", "metadata": {}, "hash": "fbd0a5c8b592fadba8b90674eb436326c6b9d909a205e7c764876ad2e3e6cc31", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2025, "end_char_idx": 4500, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from PIL import Image
import requests

img_urls =["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
           "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"]
images = [Image.open(requests.get(img_urls[0], stream=True).raw),
          Image.open(requests.get(img_urls[1], stream=True).raw)]
```

Below is an example of the chat template. We can feed conversation turns and the last message as an input by appending it at the end of the template. 


```python
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image we can see two cats on the nets."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },       
]
```

We will now call the processors' [`~ProcessorMixin.apply_chat_template`] method to preprocess its output along with the image inputs.

```python
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[images[0], images[1]], return_tensors="pt").to(device)
```

We can now pass the preprocessed inputs to the model.

```python
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)
## ['User: What do we see in this image? \nAssistant: In this image we can see two cats on the nets. \nUser: And how about this image? \nAssistant: In this image we can see flowers, plants and insect.']
```

## Streaming

We can use [text streaming](./generation_strategies#streaming) for a better generation experience. Transformers supports streaming with the [`TextStreamer`] or [`TextIteratorStreamer`] classes. We will use the [`TextIteratorStreamer`] with IDEFICS-8B.

Assume we have an application that keeps chat history and takes in the new user input. We will preprocess the inputs as usual and initialize [`TextIteratorStreamer`] to handle the generation in a separate thread. This allows you to stream the generated text tokens in real-time. Any generation arguments can be passed to [`TextIteratorStreamer`].


key: _node_content
string_value:
{"id_": "48bc4f69-dee7-495b-bcd5-565659126c20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5200b0cb-8fe5-4e61-97c4-ebc887c150aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "hash": "e9c2f2f6611fc285f25ce1b46f84a11985fff7798a572e3349601e8720a6d5de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f87c4f15-e8c6-467a-9b82-bd4e1678299d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "transformers"}, "hash": "fc27437c836012644ef191fb31ea21a13ee5e167f68bb367f5654f8d76088f1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4503, "end_char_idx": 7704, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import time
from transformers import TextIteratorStreamer
from threading import Thread

def model_inference(
    user_prompt,
    chat_history,
    max_new_tokens,
    images
):
    user_prompt = {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": user_prompt},
        ]
    }
    chat_history.append(user_prompt)
    streamer = TextIteratorStreamer(
        processor.tokenizer,
        skip_prompt=True,
        timeout=5.0,
    )

    generation_args = {
        "max_new_tokens": max_new_tokens,
        "streamer": streamer,
        "do_sample": False
    }

    # add_generation_prompt=True makes model generate bot response
    prompt = processor.apply_chat_template(chat_history, add_generation_prompt=True)
    inputs = processor(
        text=prompt,
        images=images,
        return_tensors="pt",
    ).to(device)
    generation_args.update(inputs)

    thread = Thread(
        target=model.generate,
        kwargs=generation_args,
    )
    thread.start()

    acc_text = ""
    for text_token in streamer:
        time.sleep(0.04)
        acc_text += text_token
        if acc_text.endswith("<end_of_utterance>"):
            acc_text = acc_text[:-18]
        yield acc_text
    
    thread.join()
```

Now let's call the `model_inference` function we created and stream the values. 

```python
generator = model_inference(
    user_prompt="And what is in this image?",
    chat_history=messages,
    max_new_tokens=100,
    images=images
)

for value in generator:
  print(value)

# In
# In this
# In this image ...
```

## Fit models in smaller hardware

VLMs are often large and need to be optimized to fit in smaller hardware. Transformers supports many model quantization libraries, and here we will only show int8 quantization with [Quanto](./quantization/quanto#quanto). int8 quantization offers memory improvements up to 75 percent (if all weights are quantized). However it is no free lunch, since 8-bit is not a CUDA-native precision, the weights are quantized back and forth on the fly, which adds up to latency. 

First, install dependencies.

```bash
pip install -U quanto bitsandbytes
```

To quantize a model during loading, we need to first create [`QuantoConfig`]. Then load the model as usual, but pass `quantization_config` during model initialization.

```python
from transformers import Idefics2ForConditionalGeneration, AutoTokenizer, QuantoConfig

model_id = "HuggingFaceM4/idefics2-8b"
quantization_config = QuantoConfig(weights="int8")
quantized_model = Idefics2ForConditionalGeneration.from_pretrained(model_id, device_map="cuda", quantization_config=quantization_config)
```

And that's it, we can use the model the same way with no changes. 

## Further Reading

Here are some more resources for the image-text-to-text task.

- [Image-text-to-text task page](https://huggingface.co/tasks/image-text-to-text) covers model types, use cases, datasets, and more. 
- [Vision Language Models Explained](https://huggingface.co/blog/vlms) is a blog post that covers everything about vision language models and supervised fine-tuning using [TRL](https://huggingface.co/docs/trl/en/index).


key: _node_content
string_value:
{"id_": "1f9fe395-f72e-49fc-adfd-e9b1958bd3d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "98e2e437-2985-4250-afca-54da0a4b10e4", "node_type": "1", "metadata": {}, "hash": "6a5369e3d9198abbec82957c2ef70bb255363843ea67c8b218315278721eb334", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2401, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Object detection

[[open-in-colab]]

Object detection is the computer vision task of detecting instances (such as humans, buildings, or cars) in an image. Object detection models receive an image as input and output
coordinates of the bounding boxes and associated labels of the detected objects. An image can contain multiple objects,
each with its own bounding box and a label (e.g. it can have a car and a building), and each object can
be present in different parts of an image (e.g. the image can have several cars).
This task is commonly used in autonomous driving for detecting things like pedestrians, road signs, and traffic lights.
Other applications include counting objects in images, image search, and more.

In this guide, you will learn how to:

 1. Finetune [DETR](https://huggingface.co/docs/transformers/model_doc/detr), a model that combines a convolutional
 backbone with an encoder-decoder Transformer, on the [CPPE-5](https://huggingface.co/datasets/cppe-5)
 dataset.
 2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/object-detection)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q datasets transformers accelerate timm
pip install -q -U albumentations>=1.4.5 torchmetrics pycocotools
```

You'll use 🤗 Datasets to load a dataset from the Hugging Face Hub, 🤗 Transformers to train your model,
and `albumentations` to augment the data.

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

To get started, we'll define global constants, namely the model name and image size. For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.

```py
>>> MODEL_NAME = "microsoft/conditional-detr-resnet-50"  # or "facebook/detr-resnet-50"
>>> IMAGE_SIZE = 480
```

## Load the CPPE-5 dataset

The [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.


key: _node_content
string_value:
{"id_": "98e2e437-2985-4250-afca-54da0a4b10e4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1f9fe395-f72e-49fc-adfd-e9b1958bd3d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d6f38ba8fbdfced4ca39ad1b5c30477f6e4672eeadb518e179c25fa6edc3d9b1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5b93b6a-3494-4477-a6c8-ae282a8fe7b8", "node_type": "1", "metadata": {}, "hash": "e42d4eb927780ad0d43cef68cecf1a16d035757554871ad9b7283b7aef6b443e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 993, "end_char_idx": 3318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/object-detection)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q datasets transformers accelerate timm
pip install -q -U albumentations>=1.4.5 torchmetrics pycocotools
```

You'll use 🤗 Datasets to load a dataset from the Hugging Face Hub, 🤗 Transformers to train your model,
and `albumentations` to augment the data.

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

To get started, we'll define global constants, namely the model name and image size. For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.

```py
>>> MODEL_NAME = "microsoft/conditional-detr-resnet-50"  # or "facebook/detr-resnet-50"
>>> IMAGE_SIZE = 480
```

## Load the CPPE-5 dataset

The [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.

Start by loading the dataset and creating a `validation` split from `train`:

```py
>>> from datasets import load_dataset

>>> cppe5 = load_dataset("cppe-5")

>>> if "validation" not in cppe5:
...     split = cppe5["train"].train_test_split(0.15, seed=1337)
...     cppe5["train"] = split["train"]
...     cppe5["validation"] = split["test"]

>>> cppe5
DatasetDict({
    train: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 850
    })
    test: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 29
    })
    validation: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 150
    })
})
```

You'll see that this dataset has 1000 images for train and validation sets and a test set with 29 images.

To get familiar with the data, explore what the examples look like.


key: _node_content
string_value:
{"id_": "c5b93b6a-3494-4477-a6c8-ae282a8fe7b8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98e2e437-2985-4250-afca-54da0a4b10e4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "fb5e992bfefa7a687936e33d77a88565fb9b8a01c9ba28f2c330dc418ec6161b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b02f4f10-f265-42fc-b777-d90872293f52", "node_type": "1", "metadata": {}, "hash": "d0a64fc7743877eee49195caa32541c90469e2b61e9763f75539e033815242cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1887, "end_char_idx": 4475, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.

```py
>>> MODEL_NAME = "microsoft/conditional-detr-resnet-50"  # or "facebook/detr-resnet-50"
>>> IMAGE_SIZE = 480
```

## Load the CPPE-5 dataset

The [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.

Start by loading the dataset and creating a `validation` split from `train`:

```py
>>> from datasets import load_dataset

>>> cppe5 = load_dataset("cppe-5")

>>> if "validation" not in cppe5:
...     split = cppe5["train"].train_test_split(0.15, seed=1337)
...     cppe5["train"] = split["train"]
...     cppe5["validation"] = split["test"]

>>> cppe5
DatasetDict({
    train: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 850
    })
    test: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 29
    })
    validation: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 150
    })
})
```

You'll see that this dataset has 1000 images for train and validation sets and a test set with 29 images.

To get familiar with the data, explore what the examples look like.

```py
>>> cppe5["train"][0]
{
  'image_id': 366,
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=500x290>,
  'width': 500,
  'height': 500,
  'objects': {
    'id': [1932, 1933, 1934],
    'area': [27063, 34200, 32431],
    'bbox': [[29.0, 11.0, 97.0, 279.0],
      [201.0, 1.0, 120.0, 285.0],
      [382.0, 0.0, 113.0, 287.0]],
    'category': [0, 0, 0]
  }
}
```

The examples in the dataset have the following fields:
- `image_id`: the example image id
- `image`: a `PIL.Image.Image` object containing the image
- `width`: width of the image
- `height`: height of the image
- `objects`: a dictionary containing bounding box metadata for the objects in the image:
  - `id`: the annotation id
  - `area`: the area of the bounding box
  - `bbox`: the object's bounding box (in the [COCO format](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#coco) )
  - `category`: the object's category, with possible values including `Coverall (0)`, `Face_Shield (1)`, `Gloves (2)`, `Goggles (3)` and `Mask (4)`

You may notice that the `bbox` field follows the COCO format, which is the format that the DETR model expects.


key: _node_content
string_value:
{"id_": "b02f4f10-f265-42fc-b777-d90872293f52", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5b93b6a-3494-4477-a6c8-ae282a8fe7b8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "4d4a5b58470ff6062871fbb8018f2a047431d7f0c606b47279d78be270b5ee10", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "21cb8211-b140-4bb4-a2d8-62b84dfb0cd0", "node_type": "1", "metadata": {}, "hash": "76dcc5b53250fae83d0ef8ef2392f35fbdc19b1934b867946bb7c4b697025fe5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3320, "end_char_idx": 4760, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> cppe5["train"][0]
{
  'image_id': 366,
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=500x290>,
  'width': 500,
  'height': 500,
  'objects': {
    'id': [1932, 1933, 1934],
    'area': [27063, 34200, 32431],
    'bbox': [[29.0, 11.0, 97.0, 279.0],
      [201.0, 1.0, 120.0, 285.0],
      [382.0, 0.0, 113.0, 287.0]],
    'category': [0, 0, 0]
  }
}
```

The examples in the dataset have the following fields:
- `image_id`: the example image id
- `image`: a `PIL.Image.Image` object containing the image
- `width`: width of the image
- `height`: height of the image
- `objects`: a dictionary containing bounding box metadata for the objects in the image:
  - `id`: the annotation id
  - `area`: the area of the bounding box
  - `bbox`: the object's bounding box (in the [COCO format](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#coco) )
  - `category`: the object's category, with possible values including `Coverall (0)`, `Face_Shield (1)`, `Gloves (2)`, `Goggles (3)` and `Mask (4)`

You may notice that the `bbox` field follows the COCO format, which is the format that the DETR model expects.
However, the grouping of the fields inside `objects` differs from the annotation format DETR requires. You will
need to apply some preprocessing transformations before using this data for training.

To get an even better understanding of the data, visualize an example in the dataset.


key: _node_content
string_value:
{"id_": "21cb8211-b140-4bb4-a2d8-62b84dfb0cd0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b02f4f10-f265-42fc-b777-d90872293f52", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "f1143e19fc5aefe896eeb089aa35651e4823c2f7beabacf48633fa7509579910", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "badedd7f-d690-4f73-92a6-829eef17391f", "node_type": "1", "metadata": {}, "hash": "7c29284067bf392a3b88dae6f2f99a9958154e4b108ac43616af81f2d9b5b4ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4476, "end_char_idx": 7433, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, the grouping of the fields inside `objects` differs from the annotation format DETR requires. You will
need to apply some preprocessing transformations before using this data for training.

To get an even better understanding of the data, visualize an example in the dataset.

```py
>>> import numpy as np
>>> import os
>>> from PIL import Image, ImageDraw

>>> image = cppe5["train"][2]["image"]
>>> annotations = cppe5["train"][2]["objects"]
>>> draw = ImageDraw.Draw(image)

>>> categories = cppe5["train"].features["objects"].feature["category"].names

>>> id2label = {index: x for index, x in enumerate(categories, start=0)}
>>> label2id = {v: k for k, v in id2label.items()}

>>> for i in range(len(annotations["id"])):
...     box = annotations["bbox"][i]
...     class_idx = annotations["category"][i]
...     x, y, w, h = tuple(box)
...     # Check if coordinates are normalized or not
...     if max(box) > 1.0:
...         # Coordinates are un-normalized, no need to re-scale them
...         x1, y1 = int(x), int(y)
...         x2, y2 = int(x + w), int(y + h)
...     else:
...         # Coordinates are normalized, re-scale them
...         x1 = int(x * width)
...         y1 = int(y * height)
...         x2 = int((x + w) * width)
...         y2 = int((y + h) * height)
...     draw.rectangle((x, y, x + w, y + h), outline="red", width=1)
...     draw.text((x, y), id2label[class_idx], fill="white")

>>> image
```
<div class="flex justify-center">
    <img src="https://i.imgur.com/oVQb9SF.png" alt="CPPE-5 Image Example"/>
</div>


To visualize the bounding boxes with associated labels, you can get the labels from the dataset's metadata, specifically
the `category` field.
You'll also want to create dictionaries that map a label id to a label class (`id2label`) and the other way around (`label2id`).
You can use them later when setting up the model. Including these maps will make your model reusable by others if you share
it on the Hugging Face Hub. Please note that, the part of above code that draws the bounding boxes assume that it is in `COCO` format `(x_min, y_min, width, height)`. It has to be adjusted to work for other formats like `(x_min, y_min, x_max, y_max)`.

As a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for
object detection is bounding boxes that "stretch" beyond the edge of the image. Such "runaway" bounding boxes can raise
errors during training and should be addressed. There are a few examples with this issue in this dataset.
To keep things simple in this guide, we will set `clip=True` for `BboxParams` in transformations below.

## Preprocess the data

To finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.
[`AutoImageProcessor`] takes care of processing image data to create `pixel_values`, `pixel_mask`, and
`labels` that a DETR model can train with.


key: _node_content
string_value:
{"id_": "badedd7f-d690-4f73-92a6-829eef17391f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "21cb8211-b140-4bb4-a2d8-62b84dfb0cd0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "081c3c7d56afcca8c583c9c0d912b4f336db28f1e7091f448695bf4369200b45", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad89c284-4d87-444a-89f4-bcea83f1546d", "node_type": "1", "metadata": {}, "hash": "c35e953b9512e753411c4004b1f2b62e16c042dfb529d7f77cc8dea243add8fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6033, "end_char_idx": 9114, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To visualize the bounding boxes with associated labels, you can get the labels from the dataset's metadata, specifically
the `category` field.
You'll also want to create dictionaries that map a label id to a label class (`id2label`) and the other way around (`label2id`).
You can use them later when setting up the model. Including these maps will make your model reusable by others if you share
it on the Hugging Face Hub. Please note that, the part of above code that draws the bounding boxes assume that it is in `COCO` format `(x_min, y_min, width, height)`. It has to be adjusted to work for other formats like `(x_min, y_min, x_max, y_max)`.

As a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for
object detection is bounding boxes that "stretch" beyond the edge of the image. Such "runaway" bounding boxes can raise
errors during training and should be addressed. There are a few examples with this issue in this dataset.
To keep things simple in this guide, we will set `clip=True` for `BboxParams` in transformations below.

## Preprocess the data

To finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.
[`AutoImageProcessor`] takes care of processing image data to create `pixel_values`, `pixel_mask`, and
`labels` that a DETR model can train with. The image processor has some attributes that you won't have to worry about:

- `image_mean = [0.485, 0.456, 0.406 ]`
- `image_std = [0.229, 0.224, 0.225]`

These are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial
to replicate when doing inference or finetuning a pre-trained image model.

Instantiate the image processor from the same checkpoint as the model you want to finetune.

```py
>>> from transformers import AutoImageProcessor

>>> MAX_SIZE = IMAGE_SIZE

>>> image_processor = AutoImageProcessor.from_pretrained(
...     MODEL_NAME,
...     do_resize=True,
...     size={"max_height": MAX_SIZE, "max_width": MAX_SIZE},
...     do_pad=True,
...     pad_size={"height": MAX_SIZE, "width": MAX_SIZE},
... )
```

Before passing the images to the `image_processor`, apply two preprocessing transformations to the dataset:
- Augmenting images
- Reformatting annotations to meet DETR expectations

First, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use [Albumentations](https://albumentations.ai/docs/).
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),
and it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).


key: _node_content
string_value:
{"id_": "ad89c284-4d87-444a-89f4-bcea83f1546d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "badedd7f-d690-4f73-92a6-829eef17391f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "549ff75ee709b05712c0812ff314a076a1619e8dc17cd01a0e13ed5ac7146eae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "374ec709-a61c-4e41-ac96-0582daa5a52b", "node_type": "1", "metadata": {}, "hash": "e3d3408c28ef45c5a0f556e3d1aa06cacde680bf44f90dd18b57b2524787728a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7434, "end_char_idx": 10095, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The image processor has some attributes that you won't have to worry about:

- `image_mean = [0.485, 0.456, 0.406 ]`
- `image_std = [0.229, 0.224, 0.225]`

These are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial
to replicate when doing inference or finetuning a pre-trained image model.

Instantiate the image processor from the same checkpoint as the model you want to finetune.

```py
>>> from transformers import AutoImageProcessor

>>> MAX_SIZE = IMAGE_SIZE

>>> image_processor = AutoImageProcessor.from_pretrained(
...     MODEL_NAME,
...     do_resize=True,
...     size={"max_height": MAX_SIZE, "max_width": MAX_SIZE},
...     do_pad=True,
...     pad_size={"height": MAX_SIZE, "width": MAX_SIZE},
... )
```

Before passing the images to the `image_processor`, apply two preprocessing transformations to the dataset:
- Augmenting images
- Reformatting annotations to meet DETR expectations

First, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use [Albumentations](https://albumentations.ai/docs/).
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),
and it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).

```py
>>> import albumentations as A

>>> train_augment_and_transform = A.Compose(
...     [
...         A.Perspective(p=0.1),
...         A.HorizontalFlip(p=0.5),
...         A.RandomBrightnessContrast(p=0.5),
...         A.HueSaturationValue(p=0.1),
...     ],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True, min_area=25),
... )

>>> validation_transform = A.Compose(
...     [A.NoOp()],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True),
... )
```

The `image_processor` expects the annotations to be in the following format: `{'image_id': int, 'annotations': List[Dict]}`,
 where each dictionary is a COCO object annotation. Let's add a function to reformat annotations for a single example:

```py
>>> def format_image_annotations_as_coco(image_id, categories, areas, bboxes):
...     """Format one set of image annotations to the COCO format

...     Args:
...         image_id (str): image id. e.g.


key: _node_content
string_value:
{"id_": "374ec709-a61c-4e41-ac96-0582daa5a52b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad89c284-4d87-444a-89f4-bcea83f1546d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "eb1f571d66366d801c6728c9cc06ff1d48326836ad03c9c2b6b16b9ad1c94739", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "64520fcd-832d-41d8-a937-2caa109015df", "node_type": "1", "metadata": {}, "hash": "b84914bda320b3f08e5e432afb8242660428b333d68de5ea81aa07cef3ea7ced", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8538, "end_char_idx": 12459, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here we use [Albumentations](https://albumentations.ai/docs/).
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),
and it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).

```py
>>> import albumentations as A

>>> train_augment_and_transform = A.Compose(
...     [
...         A.Perspective(p=0.1),
...         A.HorizontalFlip(p=0.5),
...         A.RandomBrightnessContrast(p=0.5),
...         A.HueSaturationValue(p=0.1),
...     ],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True, min_area=25),
... )

>>> validation_transform = A.Compose(
...     [A.NoOp()],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True),
... )
```

The `image_processor` expects the annotations to be in the following format: `{'image_id': int, 'annotations': List[Dict]}`,
 where each dictionary is a COCO object annotation. Let's add a function to reformat annotations for a single example:

```py
>>> def format_image_annotations_as_coco(image_id, categories, areas, bboxes):
...     """Format one set of image annotations to the COCO format

...     Args:
...         image_id (str): image id. e.g. "0001"
...         categories (List[int]): list of categories/class labels corresponding to provided bounding boxes
...         areas (List[float]): list of corresponding areas to provided bounding boxes
...         bboxes (List[Tuple[float]]): list of bounding boxes provided in COCO format
...             ([center_x, center_y, width, height] in absolute coordinates)

...     Returns:
...         dict: {
...             "image_id": image id,
...             "annotations": list of formatted annotations
...         }
...     """
...     annotations = []
...     for category, area, bbox in zip(categories, areas, bboxes):
...         formatted_annotation = {
...             "image_id": image_id,
...             "category_id": category,
...             "iscrowd": 0,
...             "area": area,
...             "bbox": list(bbox),
...         }
...         annotations.append(formatted_annotation)

...     return {
...         "image_id": image_id,
...         "annotations": annotations,
...     }

```

Now you can combine the image and annotation transformations to use on a batch of examples:

```py
>>> def augment_and_transform_batch(examples, transform, image_processor, return_pixel_mask=False):
...     """Apply augmentations and format annotations in COCO format for object detection task"""

...     images = []
...     annotations = []
...     for image_id, image, objects in zip(examples["image_id"], examples["image"], examples["objects"]):
...         image = np.array(image.convert("RGB"))

...         # apply augmentations
...         output = transform(image=image, bboxes=objects["bbox"], category=objects["category"])
...         images.append(output["image"])

...         # format annotations in COCO format
...         formatted_annotations = format_image_annotations_as_coco(
...             image_id, output["category"], objects["area"], output["bboxes"]
...         )
...         annotations.append(formatted_annotations)

...     # Apply the image processor transformations: resizing, rescaling, normalization
...     result = image_processor(images=images, annotations=annotations, return_tensors="pt")

...     if not return_pixel_mask:
...         result.pop("pixel_mask", None)

...     return result
```

Apply this preprocessing function to the entire dataset using 🤗 Datasets [`~datasets.Dataset.with_transform`] method.


key: _node_content
string_value:
{"id_": "64520fcd-832d-41d8-a937-2caa109015df", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "374ec709-a61c-4e41-ac96-0582daa5a52b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "fe71bd0699a05148e04169c9045199cf2337e343f4d0aa148ecb27078aa3dc53", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7249101-a2ad-4eb9-ba8d-d7d34282ec7a", "node_type": "1", "metadata": {}, "hash": "16362efe4a5f5d6d55d670328bfffcfbc9133716ef95184389c0dc299ad84d30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This method applies
transformations on the fly when you load an element of the dataset.

At this point, you can check what an example from the dataset looks like after the transformations. You should see a tensor
with `pixel_values`, a tensor with `pixel_mask`, and `labels`.

```py
>>> from functools import partial

>>> # Make transform functions for batch and apply for dataset splits
>>> train_transform_batch = partial(
.     augment_and_transform_batch, transform=train_augment_and_transform, image_processor=image_processor
. )
>>> validation_transform_batch = partial(
.     augment_and_transform_batch, transform=validation_transform, image_processor=image_processor
. )

>>> cppe5["train"] = cppe5["train"].with_transform(train_transform_batch)
>>> cppe5["validation"] = cppe5["validation"].with_transform(validation_transform_batch)
>>> cppe5["test"] = cppe5["test"].with_transform(validation_transform_batch)

>>> cppe5["train"][15]
{'pixel_values': tensor([[[ 1.9235,  1.9407,  1.9749,  . -0.7822, -0.7479, -0.6965],
          [ 1.9578,  1.9749,  1.9920,  . -0.7993, -0.7650, -0.7308],
          [ 2.0092,  2.0092,  2.0263,  . -0.8507, -0.8164, -0.7822],
          .
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741]],
  
          [[ 1.6232,  1.6408,  1.6583,  .  0.8704,  1.0105,  1.1331],
          [ 1.6408,  1.6583,  1.6758,  .  0.8529,  0.9930,  1.0980],
          [ 1.6933,  1.6933,  1.7108,  .  0.8179,  0.9580,  1.0630],
          .
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052]],
  
          [[ 1.8905,  1.9080,  1.9428,  .


key: _node_content
string_value:
{"id_": "f7249101-a2ad-4eb9-ba8d-d7d34282ec7a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "64520fcd-832d-41d8-a937-2caa109015df", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "8ecec3def1f295328ef58b900262969b60b75961d859f60bfba51f7f7367b3ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e178de5b-0b04-4669-bf51-079c387f2396", "node_type": "1", "metadata": {}, "hash": "18d08ee815e7860e2de3464b851dacbeb4a07d76cc200bc0f4f0ffa7f265a47e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741]],
  
          [[ 1.6232,  1.6408,  1.6583,  .  0.8704,  1.0105,  1.1331],
          [ 1.6408,  1.6583,  1.6758,  .  0.8529,  0.9930,  1.0980],
          [ 1.6933,  1.6933,  1.7108,  .  0.8179,  0.9580,  1.0630],
          .
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052]],
  
          [[ 1.8905,  1.9080,  1.9428,  . -0.1487, -0.0964, -0.0615],
          [ 1.9254,  1.9428,  1.9603,  . -0.1661, -0.1138, -0.0790],
          [ 1.9777,  1.9777,  1.9951,  . -0.2010, -0.1138, -0.0790],
          .
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265]]]),
  'labels': {'image_id': tensor([688]), 'class_labels': tensor([3, 4, 2, 0, 0]), 'boxes': tensor([[0.4700, 0.1933, 0.1467, 0.0767],
          [0.4858, 0.2600, 0.1150, 0.1000],
          [0.4042, 0.4517, 0.1217, 0.1300],
          [0.4242, 0.3217, 0.3617, 0.5567],


key: _node_content
string_value:
{"id_": "e178de5b-0b04-4669-bf51-079c387f2396", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7249101-a2ad-4eb9-ba8d-d7d34282ec7a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "465d8093f52c6ddf2c020f65f2241f59bc65da76bf92d35b2dded3765073efc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f20127c4-cd50-4b98-8051-6d38ffe045d6", "node_type": "1", "metadata": {}, "hash": "e58d179bc248edbe2e04351c151c238e22920aa8707ea729758619aa449148a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9080,  1.9428,  . -0.1487, -0.0964, -0.0615],
          [ 1.9254,  1.9428,  1.9603,  . -0.1661, -0.1138, -0.0790],
          [ 1.9777,  1.9777,  1.9951,  . -0.2010, -0.1138, -0.0790],
          .
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265]]]),
  'labels': {'image_id': tensor([688]), 'class_labels': tensor([3, 4, 2, 0, 0]), 'boxes': tensor([[0.4700, 0.1933, 0.1467, 0.0767],
          [0.4858, 0.2600, 0.1150, 0.1000],
          [0.4042, 0.4517, 0.1217, 0.1300],
          [0.4242, 0.3217, 0.3617, 0.5567],
          [0.6617, 0.4033, 0.5400, 0.4533]]), 'area': tensor([ 4048.  4140.  5694. 72478. 88128.]), 'iscrowd': tensor([0, 0, 0, 0, 0]), 'orig_size': tensor([480, 480])}}
```

You have successfully augmented the individual images and prepared their annotations. However, preprocessing isn't
complete yet. In the final step, create a custom `collate_fn` to batch images together.
Pad images (which are now `pixel_values`) to the largest image in a batch, and create a corresponding `pixel_mask`
to indicate which pixels are real (1) and which are padding (0).

```py
>>> import torch

>>> def collate_fn(batch):
...     data = {}
...     data["pixel_values"] = torch.stack([x["pixel_values"] for x in batch])
...     data["labels"] = [x["labels"] for x in batch]
...     if "pixel_mask" in batch[0]:
...         data["pixel_mask"] = torch.stack([x["pixel_mask"] for x in batch])
...     return data

```

## Preparing function to compute mAP

Object detection models are commonly evaluated with a set of <a href="https://cocodataset.org/#detection-eval">COCO-style metrics</a>. We are going to use `torchmetrics` to compute `mAP` (mean average precision) and `mAR` (mean average recall) metrics and will wrap it to `compute_metrics` function in order to use in [`Trainer`] for evaluation.


key: _node_content
string_value:
{"id_": "f20127c4-cd50-4b98-8051-6d38ffe045d6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e178de5b-0b04-4669-bf51-079c387f2396", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "78542935591fa9e23c7707e33976bab8c19775aae3daa42733bd8b5537a37d08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c612183d-5c60-4bff-802f-d4bbd1bb1243", "node_type": "1", "metadata": {}, "hash": "e8654c2503d832dfc38d850656525f4459f74a861f477d5a338907750ae2e01f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.1300],
          [0.4242, 0.3217, 0.3617, 0.5567],
          [0.6617, 0.4033, 0.5400, 0.4533]]), 'area': tensor([ 4048.  4140.  5694. 72478. 88128.]), 'iscrowd': tensor([0, 0, 0, 0, 0]), 'orig_size': tensor([480, 480])}}
```

You have successfully augmented the individual images and prepared their annotations. However, preprocessing isn't
complete yet. In the final step, create a custom `collate_fn` to batch images together.
Pad images (which are now `pixel_values`) to the largest image in a batch, and create a corresponding `pixel_mask`
to indicate which pixels are real (1) and which are padding (0).

```py
>>> import torch

>>> def collate_fn(batch):
...     data = {}
...     data["pixel_values"] = torch.stack([x["pixel_values"] for x in batch])
...     data["labels"] = [x["labels"] for x in batch]
...     if "pixel_mask" in batch[0]:
...         data["pixel_mask"] = torch.stack([x["pixel_mask"] for x in batch])
...     return data

```

## Preparing function to compute mAP

Object detection models are commonly evaluated with a set of <a href="https://cocodataset.org/#detection-eval">COCO-style metrics</a>. We are going to use `torchmetrics` to compute `mAP` (mean average precision) and `mAR` (mean average recall) metrics and will wrap it to `compute_metrics` function in order to use in [`Trainer`] for evaluation.

Intermediate format of boxes used for training is `YOLO` (normalized) but we will compute metrics for boxes in `Pascal VOC` (absolute) format in order to correctly handle box areas. Let's define a function that converts bounding boxes to `Pascal VOC` format:

```py
>>> from transformers.image_transforms import center_to_corners_format

>>> def convert_bbox_yolo_to_pascal(boxes, image_size):
...     """
...     Convert bounding boxes from YOLO format (x_center, y_center, width, height) in range [0, 1]
...     to Pascal VOC format (x_min, y_min, x_max, y_max) in absolute coordinates.

...     Args:
...         boxes (torch.Tensor): Bounding boxes in YOLO format
...         image_size (Tuple[int, int]): Image size in format (height, width)

...     Returns:
...         torch.Tensor: Bounding boxes in Pascal VOC format (x_min, y_min, x_max, y_max)
...     """
...     # convert center to corners format
...     boxes = center_to_corners_format(boxes)

...     # convert to absolute coordinates
...     height, width = image_size
...     boxes = boxes * torch.tensor([[width, height, width, height]])

...     return boxes
```

Then, in `compute_metrics` function we collect `predicted` and `target` bounding boxes, scores and labels from evaluation loop results and pass it to the scoring function.

```py
>>> import numpy as np
>>> from dataclasses import dataclass
>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision


>>> @dataclass
>>> class ModelOutput:
...     logits: torch.Tensor
...     pred_boxes: torch.Tensor


key: _node_content
string_value:
{"id_": "c612183d-5c60-4bff-802f-d4bbd1bb1243", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f20127c4-cd50-4b98-8051-6d38ffe045d6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "a9df26324f1551da173a70883d3e2bb812122accfe25835fb4317135a8f3fd97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c", "node_type": "1", "metadata": {}, "hash": "1ffb0299a3d99bff0a66034bb4407641898738e0867c0aa7315f300ca0ea0d4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16339, "end_char_idx": 21366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Intermediate format of boxes used for training is `YOLO` (normalized) but we will compute metrics for boxes in `Pascal VOC` (absolute) format in order to correctly handle box areas. Let's define a function that converts bounding boxes to `Pascal VOC` format:

```py
>>> from transformers.image_transforms import center_to_corners_format

>>> def convert_bbox_yolo_to_pascal(boxes, image_size):
...     """
...     Convert bounding boxes from YOLO format (x_center, y_center, width, height) in range [0, 1]
...     to Pascal VOC format (x_min, y_min, x_max, y_max) in absolute coordinates.

...     Args:
...         boxes (torch.Tensor): Bounding boxes in YOLO format
...         image_size (Tuple[int, int]): Image size in format (height, width)

...     Returns:
...         torch.Tensor: Bounding boxes in Pascal VOC format (x_min, y_min, x_max, y_max)
...     """
...     # convert center to corners format
...     boxes = center_to_corners_format(boxes)

...     # convert to absolute coordinates
...     height, width = image_size
...     boxes = boxes * torch.tensor([[width, height, width, height]])

...     return boxes
```

Then, in `compute_metrics` function we collect `predicted` and `target` bounding boxes, scores and labels from evaluation loop results and pass it to the scoring function.

```py
>>> import numpy as np
>>> from dataclasses import dataclass
>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision


>>> @dataclass
>>> class ModelOutput:
...     logits: torch.Tensor
...     pred_boxes: torch.Tensor


>>> @torch.no_grad()
>>> def compute_metrics(evaluation_results, image_processor, threshold=0.0, id2label=None):
...     """
...     Compute mean average mAP, mAR and their variants for the object detection task.

...     Args:
...         evaluation_results (EvalPrediction): Predictions and targets from evaluation.
...         threshold (float, optional): Threshold to filter predicted boxes by confidence. Defaults to 0.0.
...         id2label (Optional[dict], optional): Mapping from class id to class name. Defaults to None.

...     Returns:
...         Mapping[str, float]: Metrics in a form of dictionary {<metric_name>: <metric_value>}
...     """

...     predictions, targets = evaluation_results.predictions, evaluation_results.label_ids

...     # For metric computation we need to provide:
...     #  - targets in a form of list of dictionaries with keys "boxes", "labels"
...     #  - predictions in a form of list of dictionaries with keys "boxes", "scores", "labels"

...     image_sizes = []
...     post_processed_targets = []
...     post_processed_predictions = []

...     # Collect targets in the required format for metric computation
...     for batch in targets:
...         # collect image sizes, we will need them for predictions post processing
...         batch_image_sizes = torch.tensor(np.array([x["orig_size"] for x in batch]))
...         image_sizes.append(batch_image_sizes)
...         # collect targets in the required format for metric computation
...         # boxes were converted to YOLO format needed for model training
...         # here we will convert them to Pascal VOC format (x_min, y_min, x_max, y_max)
...         for image_target in batch:
...             boxes = torch.tensor(image_target["boxes"])
...             boxes = convert_bbox_yolo_to_pascal(boxes, image_target["orig_size"])
...             labels = torch.tensor(image_target["class_labels"])
...             post_processed_targets.append({"boxes": boxes, "labels": labels})

...     # Collect predictions in the required format for metric computation,
...     # model produce boxes in YOLO format, then image_processor convert them to Pascal VOC format
...     for batch, target_sizes in zip(predictions, image_sizes):
...         batch_logits, batch_boxes = batch[1], batch[2]
...         output = ModelOutput(logits=torch.tensor(batch_logits), pred_boxes=torch.tensor(batch_boxes))
...         post_processed_output = image_processor.post_process_object_detection(
...             output, threshold=threshold, target_sizes=target_sizes
...         )
...         post_processed_predictions.extend(post_processed_output)

...     # Compute metrics
...     metric = MeanAveragePrecision(box_format="xyxy", class_metrics=True)
...     metric.update(post_processed_predictions, post_processed_targets)
...     metrics = metric.compute()

...     # Replace list of per class metrics with separate metric for each class
...     classes = metrics.pop("classes")
...     map_per_class = metrics.pop("map_per_class")
...     mar_100_per_class = metrics.pop("mar_100_per_class")
...     for class_id, class_map, class_mar in zip(classes, map_per_class, mar_100_per_class):
...         class_name = id2label[class_id.item()] if id2label is not None else class_id.item()
...         metrics[f"map_{class_name}"] = class_map
...         metrics[f"mar_100_{class_name}"] = class_mar

...     metrics = {k: round(v.item(), 4) for k, v in metrics.items()}

...     return metrics


key: _node_content
string_value:
{"id_": "a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c612183d-5c60-4bff-802f-d4bbd1bb1243", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "55c79b23536ee0c2c5e518c276850ba6c3369decb88493b9a435605d883a70f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbc72b81-8b34-4029-baec-087f7323e1db", "node_type": "1", "metadata": {}, "hash": "203b9167d068489e551c4d0a452a1bf3986c1b59b1342a2e5024b962b79e769e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> eval_compute_metrics_fn = partial(
...     compute_metrics, image_processor=image_processor, id2label=id2label, threshold=0.0
... )
```

## Training the detection model

You have done most of the heavy lifting in the previous sections, so now you are ready to train your model!
The images in this dataset are still quite large, even after resizing. This means that finetuning this model will
require at least one GPU.

Training involves the following steps:
1. Load the model with [`AutoModelForObjectDetection`] using the same checkpoint as in the preprocessing.
2. Define your training hyperparameters in [`TrainingArguments`].
3. Pass the training arguments to [`Trainer`] along with the model, dataset, image processor, and data collator.
4. Call [`~Trainer.train`] to finetune your model.

When loading the model from the same checkpoint that you used for the preprocessing, remember to pass the `label2id`
and `id2label` maps that you created earlier from the dataset's metadata. Additionally, we specify `ignore_mismatched_sizes=True` to replace the existing classification head with a new one.

```py
>>> from transformers import AutoModelForObjectDetection

>>> model = AutoModelForObjectDetection.from_pretrained(
...     MODEL_NAME,
...     id2label=id2label,
...     label2id=label2id,
...     ignore_mismatched_sizes=True,
... )
```

In the [`TrainingArguments`] use `output_dir` to specify where to save your model, then configure hyperparameters as you see fit. For `num_train_epochs=30` training will take about 35 minutes in Google Colab T4 GPU, increase the number of epoch to get better results.

Important notes:
 - Do not remove unused columns because this will drop the image column. Without the image column, you
can't create `pixel_values`. For this reason, set `remove_unused_columns` to `False`.
 - Set `eval_do_concat_batches=False` to get proper evaluation results. Images have different number of target boxes, if batches are concatenated we will not be able to determine which boxes belongs to particular image.

If you wish to share your model by pushing to the Hub, set `push_to_hub` to `True` (you must be signed in to Hugging
Face to upload your model).

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(
.     output_dir="detr_finetuned_cppe5",
.     num_train_epochs=30,
.     fp16=False,
.     per_device_train_batch_size=8,
.     dataloader_num_workers=4,
.     learning_rate=5e-5,
.     lr_scheduler_type="cosine",
.     weight_decay=1e-4,
.     max_grad_norm=0.01,
.     metric_for_best_model="eval_map",
.     greater_is_better=True,
.     load_best_model_at_end=True,
.     eval_strategy="epoch",
.     save_strategy="epoch",
.     save_total_limit=2,
.     remove_unused_columns=False,
.     eval_do_concat_batches=False,
.     push_to_hub=True,
. )
```

Finally, bring everything together, and call [`~transformers.Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
.     model=model,
.     args=training_args,
.     train_dataset=cppe5["train"],
.     eval_dataset=cppe5["validation"],
.     tokenizer=image_processor,
.     data_collator=collate_fn,
.


key: _node_content
string_value:
{"id_": "fbc72b81-8b34-4029-baec-087f7323e1db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4c5847e-d7b4-4332-a37c-bfbf6abb4c0c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "81831528b8a21a2723008c21d15b4d039a0100f6a1874bd9ac01d5fbc40ff9c5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c48c38dc-1130-469e-b50b-eaefb02af16a", "node_type": "1", "metadata": {}, "hash": "373bc6419be3b6ecc65f9036aaa36cd93684e1f2bfa34c2b0a662440b84d8ffc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Without the image column, you
can't create `pixel_values`. For this reason, set `remove_unused_columns` to `False`.
 - Set `eval_do_concat_batches=False` to get proper evaluation results. Images have different number of target boxes, if batches are concatenated we will not be able to determine which boxes belongs to particular image.

If you wish to share your model by pushing to the Hub, set `push_to_hub` to `True` (you must be signed in to Hugging
Face to upload your model).

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(
.     output_dir="detr_finetuned_cppe5",
.     num_train_epochs=30,
.     fp16=False,
.     per_device_train_batch_size=8,
.     dataloader_num_workers=4,
.     learning_rate=5e-5,
.     lr_scheduler_type="cosine",
.     weight_decay=1e-4,
.     max_grad_norm=0.01,
.     metric_for_best_model="eval_map",
.     greater_is_better=True,
.     load_best_model_at_end=True,
.     eval_strategy="epoch",
.     save_strategy="epoch",
.     save_total_limit=2,
.     remove_unused_columns=False,
.     eval_do_concat_batches=False,
.     push_to_hub=True,
. )
```

Finally, bring everything together, and call [`~transformers.Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
.     model=model,
.     args=training_args,
.     train_dataset=cppe5["train"],
.     eval_dataset=cppe5["validation"],
.     tokenizer=image_processor,
.     data_collator=collate_fn,
.     compute_metrics=eval_compute_metrics_fn,
. )

>>> trainer.train()
```
<div>

  <progress value='3210' max='3210' style='width:300px; height:20px; vertical-align: middle;'></progress>
  [3210/3210 26:07, Epoch 30/30]
</div>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>Epoch</th>
      <th>Training Loss</th>
      <th>Validation Loss</th>
      <th>Map</th>
      <th>Map 50</th>
      <th>Map 75</th>
      <th>Map Small</th>
      <th>Map Medium</th>
      <th>Map Large</th>
      <th>Mar 1</th>
      <th>Mar 10</th>
      <th>Mar 100</th>
      <th>Mar Small</th>
      <th>Mar Medium</th>
      <th>Mar Large</th>
      <th>Map Coverall</th>
      <th>Mar 100 Coverall</th>
      <th>Map Face Shield</th>
      <th>Mar 100 Face Shield</th>
      <th>Map Gloves</th>
      <th>Mar 100 Gloves</th>
      <th>Map Goggles</th>
      <th>Mar 100 Goggles</th>
      <th>Map Mask</th>
      <th>Mar 100 Mask</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td>No log</td>
      <td>2.


key: _node_content
string_value:
{"id_": "c48c38dc-1130-469e-b50b-eaefb02af16a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbc72b81-8b34-4029-baec-087f7323e1db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "0d629bb88d97aa32173202b6aa162f0a49e0f47e4983abd4a1d4cd368a0a08ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7cf020a-5bac-4c3d-b345-b5c09465413e", "node_type": "1", "metadata": {}, "hash": "e3b8d70541fd887d0c347858c3265f25f555d6403e49d8081b16a8f7cb15ef44", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
tokenizer=image_processor,
.     data_collator=collate_fn,
.     compute_metrics=eval_compute_metrics_fn,
. )

>>> trainer.train()
```
<div>

  <progress value='3210' max='3210' style='width:300px; height:20px; vertical-align: middle;'></progress>
  [3210/3210 26:07, Epoch 30/30]
</div>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>Epoch</th>
      <th>Training Loss</th>
      <th>Validation Loss</th>
      <th>Map</th>
      <th>Map 50</th>
      <th>Map 75</th>
      <th>Map Small</th>
      <th>Map Medium</th>
      <th>Map Large</th>
      <th>Mar 1</th>
      <th>Mar 10</th>
      <th>Mar 100</th>
      <th>Mar Small</th>
      <th>Mar Medium</th>
      <th>Mar Large</th>
      <th>Map Coverall</th>
      <th>Mar 100 Coverall</th>
      <th>Map Face Shield</th>
      <th>Mar 100 Face Shield</th>
      <th>Map Gloves</th>
      <th>Mar 100 Gloves</th>
      <th>Map Goggles</th>
      <th>Mar 100 Goggles</th>
      <th>Map Mask</th>
      <th>Mar 100 Mask</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td>No log</td>
      <td>2.629903</td>
      <td>0.008900</td>
      <td>0.023200</td>
      <td>0.006500</td>
      <td>0.001300</td>
      <td>0.002800</td>
      <td>0.020500</td>
      <td>0.021500</td>
      <td>0.070400</td>
      <td>0.101400</td>
      <td>0.007600</td>
      <td>0.106200</td>
      <td>0.096100</td>
      <td>0.036700</td>
      <td>0.232000</td>
      <td>0.000300</td>
      <td>0.019000</td>
      <td>0.003900</td>
      <td>0.125400</td>
      <td>0.000100</td>
      <td>0.003100</td>
      <td>0.003500</td>
      <td>0.127600</td>
    </tr>
    <tr>
      <td>2</td>
      <td>No log</td>
      <td>3.479864</td>
      <td>0.014800</td>
      <td>0.034600</td>
      <td>0.010800</td>
      <td>0.008600</td>
      <td>0.011700</td>
      <td>0.012500</td>
      <td>0.041100</td>
      <td>0.098700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "b7cf020a-5bac-4c3d-b345-b5c09465413e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c48c38dc-1130-469e-b50b-eaefb02af16a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "29bdec96642f6985411acfd94c6564a67995ae4a9ef1b4e6a735d0a0b9c15a1c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0f2da6d2-6a43-4ccf-ac27-21359ae64502", "node_type": "1", "metadata": {}, "hash": "4ec71c4473b581d4c45912aa002df0f6d85e3e42ed94493ebe3878eb5a74afbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25657, "end_char_idx": 27381, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
629903</td>
      <td>0.008900</td>
      <td>0.023200</td>
      <td>0.006500</td>
      <td>0.001300</td>
      <td>0.002800</td>
      <td>0.020500</td>
      <td>0.021500</td>
      <td>0.070400</td>
      <td>0.101400</td>
      <td>0.007600</td>
      <td>0.106200</td>
      <td>0.096100</td>
      <td>0.036700</td>
      <td>0.232000</td>
      <td>0.000300</td>
      <td>0.019000</td>
      <td>0.003900</td>
      <td>0.125400</td>
      <td>0.000100</td>
      <td>0.003100</td>
      <td>0.003500</td>
      <td>0.127600</td>
    </tr>
    <tr>
      <td>2</td>
      <td>No log</td>
      <td>3.479864</td>
      <td>0.014800</td>
      <td>0.034600</td>
      <td>0.010800</td>
      <td>0.008600</td>
      <td>0.011700</td>
      <td>0.012500</td>
      <td>0.041100</td>
      <td>0.098700</td>
      <td>0.130000</td>
      <td>0.056000</td>
      <td>0.062200</td>
      <td>0.111900</td>
      <td>0.053500</td>
      <td>0.447300</td>
      <td>0.010600</td>
      <td>0.100000</td>
      <td>0.000200</td>
      <td>0.022800</td>
      <td>0.000100</td>
      <td>0.015400</td>
      <td>0.009700</td>
      <td>0.064400</td>
    </tr>
    <tr>
      <td>3</td>
      <td>No log</td>
      <td>2.107622</td>
      <td>0.041700</td>
      <td>0.094000</td>
      <td>0.034300</td>
      <td>0.024100</td>
      <td>0.026400</td>
      <td>0.047400</td>
      <td>0.091500</td>
      <td>0.182800</td>
      <td>0.225800</td>
      <td>0.087200</td>
      <td>0.199400</td>
      <td>0.210600</td>
      <td>0.150900</td>
      <td>0.571200</td>
      <td>0.017300</td>
      <td>0.101300</td>
      <td>0.007300</td>
      <td>0.180400</td>
      <td>0.002100</td>
      <td>0.026200</td>
      <td>0.


key: _node_content
string_value:
{"id_": "0f2da6d2-6a43-4ccf-ac27-21359ae64502", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7cf020a-5bac-4c3d-b345-b5c09465413e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "7ecc94fcd9460094a5c15af65a820f189f6e7749cb6255e3ee7144242cf1b8fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3578168c-2670-4e5e-96d4-96bf9b7a277e", "node_type": "1", "metadata": {}, "hash": "fc4422e9985a27da2d43965fa2621a509b4205d33347f809f243518a1552106f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26507, "end_char_idx": 28219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
056000</td>
      <td>0.062200</td>
      <td>0.111900</td>
      <td>0.053500</td>
      <td>0.447300</td>
      <td>0.010600</td>
      <td>0.100000</td>
      <td>0.000200</td>
      <td>0.022800</td>
      <td>0.000100</td>
      <td>0.015400</td>
      <td>0.009700</td>
      <td>0.064400</td>
    </tr>
    <tr>
      <td>3</td>
      <td>No log</td>
      <td>2.107622</td>
      <td>0.041700</td>
      <td>0.094000</td>
      <td>0.034300</td>
      <td>0.024100</td>
      <td>0.026400</td>
      <td>0.047400</td>
      <td>0.091500</td>
      <td>0.182800</td>
      <td>0.225800</td>
      <td>0.087200</td>
      <td>0.199400</td>
      <td>0.210600</td>
      <td>0.150900</td>
      <td>0.571200</td>
      <td>0.017300</td>
      <td>0.101300</td>
      <td>0.007300</td>
      <td>0.180400</td>
      <td>0.002100</td>
      <td>0.026200</td>
      <td>0.031000</td>
      <td>0.250200</td>
    </tr>
    <tr>
      <td>4</td>
      <td>No log</td>
      <td>2.031242</td>
      <td>0.055900</td>
      <td>0.120600</td>
      <td>0.046900</td>
      <td>0.013800</td>
      <td>0.038100</td>
      <td>0.090300</td>
      <td>0.105900</td>
      <td>0.225600</td>
      <td>0.266100</td>
      <td>0.130200</td>
      <td>0.228100</td>
      <td>0.330000</td>
      <td>0.191000</td>
      <td>0.572100</td>
      <td>0.010600</td>
      <td>0.157000</td>
      <td>0.014600</td>
      <td>0.235300</td>
      <td>0.001700</td>
      <td>0.052300</td>
      <td>0.061800</td>
      <td>0.313800</td>
    </tr>
    <tr>
      <td>5</td>
      <td>3.889400</td>
      <td>1.883433</td>
      <td>0.089700</td>
      <td>0.201800</td>
      <td>0.067300</td>
      <td>0.022800</td>
      <td>0.


key: _node_content
string_value:
{"id_": "3578168c-2670-4e5e-96d4-96bf9b7a277e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f2da6d2-6a43-4ccf-ac27-21359ae64502", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "f8a645dd9d35f31e40672c9f0ff20692d0fb2269dc047f8fd523fcae681f48ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "944fe527-82e7-4f59-9e87-9202be212db9", "node_type": "1", "metadata": {}, "hash": "eb4318405ff3b14377badcb4ae9417f008f8e2b1828339a774c5351b74e5dc58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27357, "end_char_idx": 29071, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
026200</td>
      <td>0.031000</td>
      <td>0.250200</td>
    </tr>
    <tr>
      <td>4</td>
      <td>No log</td>
      <td>2.031242</td>
      <td>0.055900</td>
      <td>0.120600</td>
      <td>0.046900</td>
      <td>0.013800</td>
      <td>0.038100</td>
      <td>0.090300</td>
      <td>0.105900</td>
      <td>0.225600</td>
      <td>0.266100</td>
      <td>0.130200</td>
      <td>0.228100</td>
      <td>0.330000</td>
      <td>0.191000</td>
      <td>0.572100</td>
      <td>0.010600</td>
      <td>0.157000</td>
      <td>0.014600</td>
      <td>0.235300</td>
      <td>0.001700</td>
      <td>0.052300</td>
      <td>0.061800</td>
      <td>0.313800</td>
    </tr>
    <tr>
      <td>5</td>
      <td>3.889400</td>
      <td>1.883433</td>
      <td>0.089700</td>
      <td>0.201800</td>
      <td>0.067300</td>
      <td>0.022800</td>
      <td>0.065300</td>
      <td>0.129500</td>
      <td>0.136000</td>
      <td>0.272200</td>
      <td>0.303700</td>
      <td>0.112900</td>
      <td>0.312500</td>
      <td>0.424600</td>
      <td>0.300200</td>
      <td>0.585100</td>
      <td>0.032700</td>
      <td>0.202500</td>
      <td>0.031300</td>
      <td>0.271000</td>
      <td>0.008700</td>
      <td>0.126200</td>
      <td>0.075500</td>
      <td>0.333800</td>
    </tr>
    <tr>
      <td>6</td>
      <td>3.889400</td>
      <td>1.807503</td>
      <td>0.118500</td>
      <td>0.270900</td>
      <td>0.090200</td>
      <td>0.034900</td>
      <td>0.076700</td>
      <td>0.152500</td>
      <td>0.146100</td>
      <td>0.297800</td>
      <td>0.325400</td>
      <td>0.171700</td>
      <td>0.283700</td>
      <td>0.545900</td>
      <td>0.396900</td>
      <td>0.554500</td>
      <td>0.


key: _node_content
string_value:
{"id_": "944fe527-82e7-4f59-9e87-9202be212db9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3578168c-2670-4e5e-96d4-96bf9b7a277e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "6586ea3cc7637c9ab7e2d02ff475a2d700cc946b3eb7378d3e283ef73d165b48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cca2048b-99a2-4713-beee-1cb7e316f37b", "node_type": "1", "metadata": {}, "hash": "e7a715484b69a165f548bd7622a287d74ab1d22ef8ff711aba00d1860da097e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28219, "end_char_idx": 29935, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
065300</td>
      <td>0.129500</td>
      <td>0.136000</td>
      <td>0.272200</td>
      <td>0.303700</td>
      <td>0.112900</td>
      <td>0.312500</td>
      <td>0.424600</td>
      <td>0.300200</td>
      <td>0.585100</td>
      <td>0.032700</td>
      <td>0.202500</td>
      <td>0.031300</td>
      <td>0.271000</td>
      <td>0.008700</td>
      <td>0.126200</td>
      <td>0.075500</td>
      <td>0.333800</td>
    </tr>
    <tr>
      <td>6</td>
      <td>3.889400</td>
      <td>1.807503</td>
      <td>0.118500</td>
      <td>0.270900</td>
      <td>0.090200</td>
      <td>0.034900</td>
      <td>0.076700</td>
      <td>0.152500</td>
      <td>0.146100</td>
      <td>0.297800</td>
      <td>0.325400</td>
      <td>0.171700</td>
      <td>0.283700</td>
      <td>0.545900</td>
      <td>0.396900</td>
      <td>0.554500</td>
      <td>0.043000</td>
      <td>0.262000</td>
      <td>0.054500</td>
      <td>0.271900</td>
      <td>0.020300</td>
      <td>0.230800</td>
      <td>0.077600</td>
      <td>0.308000</td>
    </tr>
    <tr>
      <td>7</td>
      <td>3.889400</td>
      <td>1.716169</td>
      <td>0.143500</td>
      <td>0.307700</td>
      <td>0.123200</td>
      <td>0.045800</td>
      <td>0.097800</td>
      <td>0.258300</td>
      <td>0.165300</td>
      <td>0.327700</td>
      <td>0.352600</td>
      <td>0.140900</td>
      <td>0.336700</td>
      <td>0.599400</td>
      <td>0.442900</td>
      <td>0.620700</td>
      <td>0.069400</td>
      <td>0.301300</td>
      <td>0.081600</td>
      <td>0.292000</td>
      <td>0.011000</td>
      <td>0.230800</td>
      <td>0.112700</td>
      <td>0.318200</td>
    </tr>
    <tr>
      <td>8</td>
      <td>3.889400</td>
      <td>1.


key: _node_content
string_value:
{"id_": "cca2048b-99a2-4713-beee-1cb7e316f37b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "944fe527-82e7-4f59-9e87-9202be212db9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "425cad898c31c38f09ee2f2bb7c27bfa1403b5f4f27df15c126a209f92817fe9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8242a0e-91e6-4551-8317-6e89b48cbd7e", "node_type": "1", "metadata": {}, "hash": "e32bf384f8ca4d2ff6407564725c87abc278c324dfdc34dd44ad639f3b7dba9a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29071, "end_char_idx": 30787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
043000</td>
      <td>0.262000</td>
      <td>0.054500</td>
      <td>0.271900</td>
      <td>0.020300</td>
      <td>0.230800</td>
      <td>0.077600</td>
      <td>0.308000</td>
    </tr>
    <tr>
      <td>7</td>
      <td>3.889400</td>
      <td>1.716169</td>
      <td>0.143500</td>
      <td>0.307700</td>
      <td>0.123200</td>
      <td>0.045800</td>
      <td>0.097800</td>
      <td>0.258300</td>
      <td>0.165300</td>
      <td>0.327700</td>
      <td>0.352600</td>
      <td>0.140900</td>
      <td>0.336700</td>
      <td>0.599400</td>
      <td>0.442900</td>
      <td>0.620700</td>
      <td>0.069400</td>
      <td>0.301300</td>
      <td>0.081600</td>
      <td>0.292000</td>
      <td>0.011000</td>
      <td>0.230800</td>
      <td>0.112700</td>
      <td>0.318200</td>
    </tr>
    <tr>
      <td>8</td>
      <td>3.889400</td>
      <td>1.679014</td>
      <td>0.153000</td>
      <td>0.355800</td>
      <td>0.127900</td>
      <td>0.038700</td>
      <td>0.115600</td>
      <td>0.291600</td>
      <td>0.176000</td>
      <td>0.322500</td>
      <td>0.349700</td>
      <td>0.135600</td>
      <td>0.326100</td>
      <td>0.643700</td>
      <td>0.431700</td>
      <td>0.582900</td>
      <td>0.069800</td>
      <td>0.265800</td>
      <td>0.088600</td>
      <td>0.274600</td>
      <td>0.028300</td>
      <td>0.280000</td>
      <td>0.146700</td>
      <td>0.345300</td>
    </tr>
    <tr>
      <td>9</td>
      <td>3.889400</td>
      <td>1.618239</td>
      <td>0.172100</td>
      <td>0.375300</td>
      <td>0.137600</td>
      <td>0.046100</td>
      <td>0.141700</td>
      <td>0.308500</td>
      <td>0.194000</td>
      <td>0.356200</td>
      <td>0.386200</td>
      <td>0.


key: _node_content
string_value:
{"id_": "e8242a0e-91e6-4551-8317-6e89b48cbd7e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cca2048b-99a2-4713-beee-1cb7e316f37b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "f8b5cdf6200e1e523386d947330720ddec93937dd77ec3c71ebf20abaac03d8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5139a966-bcff-4638-a848-723a85a17b35", "node_type": "1", "metadata": {}, "hash": "bbbcf3788fd8f7266a224ac316f989ff78479d775dbae48d943664fd1418b997", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29935, "end_char_idx": 31640, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
679014</td>
      <td>0.153000</td>
      <td>0.355800</td>
      <td>0.127900</td>
      <td>0.038700</td>
      <td>0.115600</td>
      <td>0.291600</td>
      <td>0.176000</td>
      <td>0.322500</td>
      <td>0.349700</td>
      <td>0.135600</td>
      <td>0.326100</td>
      <td>0.643700</td>
      <td>0.431700</td>
      <td>0.582900</td>
      <td>0.069800</td>
      <td>0.265800</td>
      <td>0.088600</td>
      <td>0.274600</td>
      <td>0.028300</td>
      <td>0.280000</td>
      <td>0.146700</td>
      <td>0.345300</td>
    </tr>
    <tr>
      <td>9</td>
      <td>3.889400</td>
      <td>1.618239</td>
      <td>0.172100</td>
      <td>0.375300</td>
      <td>0.137600</td>
      <td>0.046100</td>
      <td>0.141700</td>
      <td>0.308500</td>
      <td>0.194000</td>
      <td>0.356200</td>
      <td>0.386200</td>
      <td>0.162400</td>
      <td>0.359200</td>
      <td>0.677700</td>
      <td>0.469800</td>
      <td>0.623900</td>
      <td>0.102100</td>
      <td>0.317700</td>
      <td>0.099100</td>
      <td>0.290200</td>
      <td>0.029300</td>
      <td>0.335400</td>
      <td>0.160200</td>
      <td>0.364000</td>
    </tr>
    <tr>
      <td>10</td>
      <td>1.599700</td>
      <td>1.572512</td>
      <td>0.179500</td>
      <td>0.400400</td>
      <td>0.147200</td>
      <td>0.056500</td>
      <td>0.141700</td>
      <td>0.316700</td>
      <td>0.213100</td>
      <td>0.357600</td>
      <td>0.381300</td>
      <td>0.197900</td>
      <td>0.344300</td>
      <td>0.638500</td>
      <td>0.466900</td>
      <td>0.623900</td>
      <td>0.101300</td>
      <td>0.311400</td>
      <td>0.104700</td>
      <td>0.279500</td>
      <td>0.051600</td>
      <td>0.


key: _node_content
string_value:
{"id_": "5139a966-bcff-4638-a848-723a85a17b35", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8242a0e-91e6-4551-8317-6e89b48cbd7e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "5d7ff6def374f467033c6de4a09e366aeaf661e2fc6cc1abcbe6465696ffed1e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b99d7b2-f95f-4adb-80f1-077ad8f5c75d", "node_type": "1", "metadata": {}, "hash": "939d090f4a93794149c0061a726581a0762304442ea1b3e64071ab9ec56bb0ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30787, "end_char_idx": 32506, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
162400</td>
      <td>0.359200</td>
      <td>0.677700</td>
      <td>0.469800</td>
      <td>0.623900</td>
      <td>0.102100</td>
      <td>0.317700</td>
      <td>0.099100</td>
      <td>0.290200</td>
      <td>0.029300</td>
      <td>0.335400</td>
      <td>0.160200</td>
      <td>0.364000</td>
    </tr>
    <tr>
      <td>10</td>
      <td>1.599700</td>
      <td>1.572512</td>
      <td>0.179500</td>
      <td>0.400400</td>
      <td>0.147200</td>
      <td>0.056500</td>
      <td>0.141700</td>
      <td>0.316700</td>
      <td>0.213100</td>
      <td>0.357600</td>
      <td>0.381300</td>
      <td>0.197900</td>
      <td>0.344300</td>
      <td>0.638500</td>
      <td>0.466900</td>
      <td>0.623900</td>
      <td>0.101300</td>
      <td>0.311400</td>
      <td>0.104700</td>
      <td>0.279500</td>
      <td>0.051600</td>
      <td>0.338500</td>
      <td>0.173000</td>
      <td>0.353300</td>
    </tr>
    <tr>
      <td>11</td>
      <td>1.599700</td>
      <td>1.528889</td>
      <td>0.192200</td>
      <td>0.415000</td>
      <td>0.160800</td>
      <td>0.053700</td>
      <td>0.150500</td>
      <td>0.378000</td>
      <td>0.211500</td>
      <td>0.371700</td>
      <td>0.397800</td>
      <td>0.204900</td>
      <td>0.374600</td>
      <td>0.684800</td>
      <td>0.491900</td>
      <td>0.632400</td>
      <td>0.131200</td>
      <td>0.346800</td>
      <td>0.122000</td>
      <td>0.300900</td>
      <td>0.038400</td>
      <td>0.344600</td>
      <td>0.177500</td>
      <td>0.364400</td>
    </tr>
    <tr>
      <td>12</td>
      <td>1.599700</td>
      <td>1.517532</td>
      <td>0.198300</td>
      <td>0.429800</td>
      <td>0.159800</td>
      <td>0.066400</td>
      <td>0.


key: _node_content
string_value:
{"id_": "9b99d7b2-f95f-4adb-80f1-077ad8f5c75d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5139a966-bcff-4638-a848-723a85a17b35", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "e12902a74f2ec1caecd348c00e69df5cab942ac197bae3967d92f2a91a494b3b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15a9d3b0-e041-4419-9698-29f063b94b5d", "node_type": "1", "metadata": {}, "hash": "2b8b833c8606730771ccb0e973478e78a34ee844a105af9d063206d9bb2ff453", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31640, "end_char_idx": 33359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
338500</td>
      <td>0.173000</td>
      <td>0.353300</td>
    </tr>
    <tr>
      <td>11</td>
      <td>1.599700</td>
      <td>1.528889</td>
      <td>0.192200</td>
      <td>0.415000</td>
      <td>0.160800</td>
      <td>0.053700</td>
      <td>0.150500</td>
      <td>0.378000</td>
      <td>0.211500</td>
      <td>0.371700</td>
      <td>0.397800</td>
      <td>0.204900</td>
      <td>0.374600</td>
      <td>0.684800</td>
      <td>0.491900</td>
      <td>0.632400</td>
      <td>0.131200</td>
      <td>0.346800</td>
      <td>0.122000</td>
      <td>0.300900</td>
      <td>0.038400</td>
      <td>0.344600</td>
      <td>0.177500</td>
      <td>0.364400</td>
    </tr>
    <tr>
      <td>12</td>
      <td>1.599700</td>
      <td>1.517532</td>
      <td>0.198300</td>
      <td>0.429800</td>
      <td>0.159800</td>
      <td>0.066400</td>
      <td>0.162900</td>
      <td>0.383300</td>
      <td>0.220700</td>
      <td>0.382100</td>
      <td>0.405400</td>
      <td>0.214800</td>
      <td>0.383200</td>
      <td>0.672900</td>
      <td>0.469000</td>
      <td>0.610400</td>
      <td>0.167800</td>
      <td>0.379700</td>
      <td>0.119700</td>
      <td>0.307100</td>
      <td>0.038100</td>
      <td>0.335400</td>
      <td>0.196800</td>
      <td>0.394200</td>
    </tr>
    <tr>
      <td>13</td>
      <td>1.599700</td>
      <td>1.488849</td>
      <td>0.209800</td>
      <td>0.452300</td>
      <td>0.172300</td>
      <td>0.094900</td>
      <td>0.171100</td>
      <td>0.437800</td>
      <td>0.222000</td>
      <td>0.379800</td>
      <td>0.411500</td>
      <td>0.203800</td>
      <td>0.397300</td>
      <td>0.707500</td>
      <td>0.470700</td>
      <td>0.620700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "15a9d3b0-e041-4419-9698-29f063b94b5d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b99d7b2-f95f-4adb-80f1-077ad8f5c75d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d6dd8db7830b29a1c00df414b4142ea804e01820af5b1d78cf18c4ed3df79928", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5d53a123-fccb-475c-906d-b2c3be134d66", "node_type": "1", "metadata": {}, "hash": "681df8a3bdec3ada79e24e702726a5f7b23eea3db9c4584c55c2c40b49957d62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32506, "end_char_idx": 34225, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
162900</td>
      <td>0.383300</td>
      <td>0.220700</td>
      <td>0.382100</td>
      <td>0.405400</td>
      <td>0.214800</td>
      <td>0.383200</td>
      <td>0.672900</td>
      <td>0.469000</td>
      <td>0.610400</td>
      <td>0.167800</td>
      <td>0.379700</td>
      <td>0.119700</td>
      <td>0.307100</td>
      <td>0.038100</td>
      <td>0.335400</td>
      <td>0.196800</td>
      <td>0.394200</td>
    </tr>
    <tr>
      <td>13</td>
      <td>1.599700</td>
      <td>1.488849</td>
      <td>0.209800</td>
      <td>0.452300</td>
      <td>0.172300</td>
      <td>0.094900</td>
      <td>0.171100</td>
      <td>0.437800</td>
      <td>0.222000</td>
      <td>0.379800</td>
      <td>0.411500</td>
      <td>0.203800</td>
      <td>0.397300</td>
      <td>0.707500</td>
      <td>0.470700</td>
      <td>0.620700</td>
      <td>0.186900</td>
      <td>0.407600</td>
      <td>0.124200</td>
      <td>0.306700</td>
      <td>0.059300</td>
      <td>0.355400</td>
      <td>0.207700</td>
      <td>0.367100</td>
    </tr>
    <tr>
      <td>14</td>
      <td>1.599700</td>
      <td>1.482210</td>
      <td>0.228900</td>
      <td>0.482600</td>
      <td>0.187800</td>
      <td>0.083600</td>
      <td>0.191800</td>
      <td>0.444100</td>
      <td>0.225900</td>
      <td>0.376900</td>
      <td>0.407400</td>
      <td>0.182500</td>
      <td>0.384800</td>
      <td>0.700600</td>
      <td>0.512100</td>
      <td>0.640100</td>
      <td>0.175000</td>
      <td>0.363300</td>
      <td>0.144300</td>
      <td>0.300000</td>
      <td>0.083100</td>
      <td>0.363100</td>
      <td>0.229900</td>
      <td>0.370700</td>
    </tr>
    <tr>
      <td>15</td>
      <td>1.326800</td>
      <td>1.


key: _node_content
string_value:
{"id_": "5d53a123-fccb-475c-906d-b2c3be134d66", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15a9d3b0-e041-4419-9698-29f063b94b5d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "7e76842df688ee38d98d8435e99e2edc7bb2ffe61a0d30e4a9268d5c9e44198e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb831dd5-b3cd-4c68-8173-9b9a3fd76f6e", "node_type": "1", "metadata": {}, "hash": "50120ce80d54abad8b1b009e80b1e2af3143afb5c900b75966952e9dd1a78067", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33359, "end_char_idx": 35078, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
186900</td>
      <td>0.407600</td>
      <td>0.124200</td>
      <td>0.306700</td>
      <td>0.059300</td>
      <td>0.355400</td>
      <td>0.207700</td>
      <td>0.367100</td>
    </tr>
    <tr>
      <td>14</td>
      <td>1.599700</td>
      <td>1.482210</td>
      <td>0.228900</td>
      <td>0.482600</td>
      <td>0.187800</td>
      <td>0.083600</td>
      <td>0.191800</td>
      <td>0.444100</td>
      <td>0.225900</td>
      <td>0.376900</td>
      <td>0.407400</td>
      <td>0.182500</td>
      <td>0.384800</td>
      <td>0.700600</td>
      <td>0.512100</td>
      <td>0.640100</td>
      <td>0.175000</td>
      <td>0.363300</td>
      <td>0.144300</td>
      <td>0.300000</td>
      <td>0.083100</td>
      <td>0.363100</td>
      <td>0.229900</td>
      <td>0.370700</td>
    </tr>
    <tr>
      <td>15</td>
      <td>1.326800</td>
      <td>1.475198</td>
      <td>0.216300</td>
      <td>0.455600</td>
      <td>0.174900</td>
      <td>0.088500</td>
      <td>0.183500</td>
      <td>0.424400</td>
      <td>0.226900</td>
      <td>0.373400</td>
      <td>0.404300</td>
      <td>0.199200</td>
      <td>0.396400</td>
      <td>0.677800</td>
      <td>0.496300</td>
      <td>0.633800</td>
      <td>0.166300</td>
      <td>0.392400</td>
      <td>0.128900</td>
      <td>0.312900</td>
      <td>0.085200</td>
      <td>0.312300</td>
      <td>0.205000</td>
      <td>0.370200</td>
    </tr>
    <tr>
      <td>16</td>
      <td>1.326800</td>
      <td>1.459697</td>
      <td>0.233200</td>
      <td>0.504200</td>
      <td>0.192200</td>
      <td>0.096000</td>
      <td>0.202000</td>
      <td>0.430800</td>
      <td>0.239100</td>
      <td>0.382400</td>
      <td>0.412600</td>
      <td>0.


key: _node_content
string_value:
{"id_": "bb831dd5-b3cd-4c68-8173-9b9a3fd76f6e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5d53a123-fccb-475c-906d-b2c3be134d66", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d9d9a93ee46df656774d68d037c47f113ce64c4da171378663fe30631847dd55", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c803e35-32d6-4356-aac6-a81cc44f8d63", "node_type": "1", "metadata": {}, "hash": "884bb5e26c6e24aa4a56f35a3f1cdd9b23675715e65229abf6f9129d97122a8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34225, "end_char_idx": 35931, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
475198</td>
      <td>0.216300</td>
      <td>0.455600</td>
      <td>0.174900</td>
      <td>0.088500</td>
      <td>0.183500</td>
      <td>0.424400</td>
      <td>0.226900</td>
      <td>0.373400</td>
      <td>0.404300</td>
      <td>0.199200</td>
      <td>0.396400</td>
      <td>0.677800</td>
      <td>0.496300</td>
      <td>0.633800</td>
      <td>0.166300</td>
      <td>0.392400</td>
      <td>0.128900</td>
      <td>0.312900</td>
      <td>0.085200</td>
      <td>0.312300</td>
      <td>0.205000</td>
      <td>0.370200</td>
    </tr>
    <tr>
      <td>16</td>
      <td>1.326800</td>
      <td>1.459697</td>
      <td>0.233200</td>
      <td>0.504200</td>
      <td>0.192200</td>
      <td>0.096000</td>
      <td>0.202000</td>
      <td>0.430800</td>
      <td>0.239100</td>
      <td>0.382400</td>
      <td>0.412600</td>
      <td>0.219500</td>
      <td>0.403100</td>
      <td>0.670400</td>
      <td>0.485200</td>
      <td>0.625200</td>
      <td>0.196500</td>
      <td>0.410100</td>
      <td>0.135700</td>
      <td>0.299600</td>
      <td>0.123100</td>
      <td>0.356900</td>
      <td>0.225300</td>
      <td>0.371100</td>
    </tr>
    <tr>
      <td>17</td>
      <td>1.326800</td>
      <td>1.407340</td>
      <td>0.243400</td>
      <td>0.511900</td>
      <td>0.204500</td>
      <td>0.121000</td>
      <td>0.215700</td>
      <td>0.468000</td>
      <td>0.246200</td>
      <td>0.394600</td>
      <td>0.424200</td>
      <td>0.225900</td>
      <td>0.416100</td>
      <td>0.705200</td>
      <td>0.494900</td>
      <td>0.638300</td>
      <td>0.224900</td>
      <td>0.430400</td>
      <td>0.157200</td>
      <td>0.317900</td>
      <td>0.115700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "8c803e35-32d6-4356-aac6-a81cc44f8d63", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb831dd5-b3cd-4c68-8173-9b9a3fd76f6e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "a7ba4a519846a974aaaf5ac4735e172f61ac665b886d0007c5b27635ff13ce3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f26c907-5b60-486b-8540-b379c09abc2e", "node_type": "1", "metadata": {}, "hash": "f5e2ed09e692af5f24b4ab4ac43a5967511e83d14a34b0d9c5badb2179327433", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35078, "end_char_idx": 36797, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
219500</td>
      <td>0.403100</td>
      <td>0.670400</td>
      <td>0.485200</td>
      <td>0.625200</td>
      <td>0.196500</td>
      <td>0.410100</td>
      <td>0.135700</td>
      <td>0.299600</td>
      <td>0.123100</td>
      <td>0.356900</td>
      <td>0.225300</td>
      <td>0.371100</td>
    </tr>
    <tr>
      <td>17</td>
      <td>1.326800</td>
      <td>1.407340</td>
      <td>0.243400</td>
      <td>0.511900</td>
      <td>0.204500</td>
      <td>0.121000</td>
      <td>0.215700</td>
      <td>0.468000</td>
      <td>0.246200</td>
      <td>0.394600</td>
      <td>0.424200</td>
      <td>0.225900</td>
      <td>0.416100</td>
      <td>0.705200</td>
      <td>0.494900</td>
      <td>0.638300</td>
      <td>0.224900</td>
      <td>0.430400</td>
      <td>0.157200</td>
      <td>0.317900</td>
      <td>0.115700</td>
      <td>0.369200</td>
      <td>0.224200</td>
      <td>0.365300</td>
    </tr>
    <tr>
      <td>18</td>
      <td>1.326800</td>
      <td>1.419522</td>
      <td>0.245100</td>
      <td>0.521500</td>
      <td>0.210000</td>
      <td>0.116100</td>
      <td>0.211500</td>
      <td>0.489900</td>
      <td>0.255400</td>
      <td>0.391600</td>
      <td>0.419700</td>
      <td>0.198800</td>
      <td>0.421200</td>
      <td>0.701400</td>
      <td>0.501800</td>
      <td>0.634200</td>
      <td>0.226700</td>
      <td>0.410100</td>
      <td>0.154400</td>
      <td>0.321400</td>
      <td>0.105900</td>
      <td>0.352300</td>
      <td>0.236700</td>
      <td>0.380400</td>
    </tr>
    <tr>
      <td>19</td>
      <td>1.158600</td>
      <td>1.398764</td>
      <td>0.253600</td>
      <td>0.519200</td>
      <td>0.213600</td>
      <td>0.135200</td>
      <td>0.


key: _node_content
string_value:
{"id_": "6f26c907-5b60-486b-8540-b379c09abc2e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c803e35-32d6-4356-aac6-a81cc44f8d63", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "98d410a89045fb493d4637349eaedc3477fb644feef5629736e3b1a5ef818b68", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d9a42c1f-d60b-453f-ba10-dadd1e4fe59f", "node_type": "1", "metadata": {}, "hash": "092fa7f1d5b14d0cc6a0e83453021b2e9a7d89626975ee1f414c66738f01bcae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35931, "end_char_idx": 37650, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
369200</td>
      <td>0.224200</td>
      <td>0.365300</td>
    </tr>
    <tr>
      <td>18</td>
      <td>1.326800</td>
      <td>1.419522</td>
      <td>0.245100</td>
      <td>0.521500</td>
      <td>0.210000</td>
      <td>0.116100</td>
      <td>0.211500</td>
      <td>0.489900</td>
      <td>0.255400</td>
      <td>0.391600</td>
      <td>0.419700</td>
      <td>0.198800</td>
      <td>0.421200</td>
      <td>0.701400</td>
      <td>0.501800</td>
      <td>0.634200</td>
      <td>0.226700</td>
      <td>0.410100</td>
      <td>0.154400</td>
      <td>0.321400</td>
      <td>0.105900</td>
      <td>0.352300</td>
      <td>0.236700</td>
      <td>0.380400</td>
    </tr>
    <tr>
      <td>19</td>
      <td>1.158600</td>
      <td>1.398764</td>
      <td>0.253600</td>
      <td>0.519200</td>
      <td>0.213600</td>
      <td>0.135200</td>
      <td>0.207700</td>
      <td>0.491900</td>
      <td>0.257300</td>
      <td>0.397300</td>
      <td>0.428000</td>
      <td>0.241400</td>
      <td>0.401800</td>
      <td>0.703500</td>
      <td>0.509700</td>
      <td>0.631100</td>
      <td>0.236700</td>
      <td>0.441800</td>
      <td>0.155900</td>
      <td>0.330800</td>
      <td>0.128100</td>
      <td>0.352300</td>
      <td>0.237500</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>20</td>
      <td>1.158600</td>
      <td>1.390591</td>
      <td>0.248800</td>
      <td>0.520200</td>
      <td>0.216600</td>
      <td>0.127500</td>
      <td>0.211400</td>
      <td>0.471900</td>
      <td>0.258300</td>
      <td>0.407000</td>
      <td>0.429100</td>
      <td>0.240300</td>
      <td>0.407600</td>
      <td>0.708500</td>
      <td>0.505800</td>
      <td>0.623400</td>
      <td>0.


key: _node_content
string_value:
{"id_": "d9a42c1f-d60b-453f-ba10-dadd1e4fe59f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f26c907-5b60-486b-8540-b379c09abc2e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "b9806917656863919ffcb2502f15697d67ad987f093f5e502df2a609752a3c1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0cc62b0a-741b-42f4-9780-7cf5deb4ba96", "node_type": "1", "metadata": {}, "hash": "c206e2aba1c2e2db8e5ed8f34faceb2af3c9591ce2587612691b81e43049eeb6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36797, "end_char_idx": 38516, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
207700</td>
      <td>0.491900</td>
      <td>0.257300</td>
      <td>0.397300</td>
      <td>0.428000</td>
      <td>0.241400</td>
      <td>0.401800</td>
      <td>0.703500</td>
      <td>0.509700</td>
      <td>0.631100</td>
      <td>0.236700</td>
      <td>0.441800</td>
      <td>0.155900</td>
      <td>0.330800</td>
      <td>0.128100</td>
      <td>0.352300</td>
      <td>0.237500</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>20</td>
      <td>1.158600</td>
      <td>1.390591</td>
      <td>0.248800</td>
      <td>0.520200</td>
      <td>0.216600</td>
      <td>0.127500</td>
      <td>0.211400</td>
      <td>0.471900</td>
      <td>0.258300</td>
      <td>0.407000</td>
      <td>0.429100</td>
      <td>0.240300</td>
      <td>0.407600</td>
      <td>0.708500</td>
      <td>0.505800</td>
      <td>0.623400</td>
      <td>0.235500</td>
      <td>0.431600</td>
      <td>0.150000</td>
      <td>0.325000</td>
      <td>0.125700</td>
      <td>0.375400</td>
      <td>0.227200</td>
      <td>0.390200</td>
    </tr>
    <tr>
      <td>21</td>
      <td>1.158600</td>
      <td>1.360608</td>
      <td>0.262700</td>
      <td>0.544800</td>
      <td>0.222100</td>
      <td>0.134700</td>
      <td>0.230000</td>
      <td>0.487500</td>
      <td>0.269500</td>
      <td>0.413300</td>
      <td>0.436300</td>
      <td>0.236200</td>
      <td>0.419100</td>
      <td>0.709300</td>
      <td>0.514100</td>
      <td>0.637400</td>
      <td>0.257200</td>
      <td>0.450600</td>
      <td>0.165100</td>
      <td>0.338400</td>
      <td>0.139400</td>
      <td>0.372300</td>
      <td>0.237700</td>
      <td>0.382700</td>
    </tr>
    <tr>
      <td>22</td>
      <td>1.158600</td>
      <td>1.


key: _node_content
string_value:
{"id_": "0cc62b0a-741b-42f4-9780-7cf5deb4ba96", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d9a42c1f-d60b-453f-ba10-dadd1e4fe59f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "75efab553a388d89495a58cfd25bc7a64ecdd9fba4e52092b5204803b1220211", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a42d618-dd15-4e93-a637-4ee372a7bb08", "node_type": "1", "metadata": {}, "hash": "cb90b7d5f837b23c2bb070ff668090694bd9b4d301e1303d948b919f0020fe66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37650, "end_char_idx": 39369, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
235500</td>
      <td>0.431600</td>
      <td>0.150000</td>
      <td>0.325000</td>
      <td>0.125700</td>
      <td>0.375400</td>
      <td>0.227200</td>
      <td>0.390200</td>
    </tr>
    <tr>
      <td>21</td>
      <td>1.158600</td>
      <td>1.360608</td>
      <td>0.262700</td>
      <td>0.544800</td>
      <td>0.222100</td>
      <td>0.134700</td>
      <td>0.230000</td>
      <td>0.487500</td>
      <td>0.269500</td>
      <td>0.413300</td>
      <td>0.436300</td>
      <td>0.236200</td>
      <td>0.419100</td>
      <td>0.709300</td>
      <td>0.514100</td>
      <td>0.637400</td>
      <td>0.257200</td>
      <td>0.450600</td>
      <td>0.165100</td>
      <td>0.338400</td>
      <td>0.139400</td>
      <td>0.372300</td>
      <td>0.237700</td>
      <td>0.382700</td>
    </tr>
    <tr>
      <td>22</td>
      <td>1.158600</td>
      <td>1.368296</td>
      <td>0.262800</td>
      <td>0.542400</td>
      <td>0.236400</td>
      <td>0.137400</td>
      <td>0.228100</td>
      <td>0.498500</td>
      <td>0.266500</td>
      <td>0.409000</td>
      <td>0.433000</td>
      <td>0.239900</td>
      <td>0.418500</td>
      <td>0.697500</td>
      <td>0.520500</td>
      <td>0.641000</td>
      <td>0.257500</td>
      <td>0.455700</td>
      <td>0.162600</td>
      <td>0.334800</td>
      <td>0.140200</td>
      <td>0.353800</td>
      <td>0.233200</td>
      <td>0.379600</td>
    </tr>
    <tr>
      <td>23</td>
      <td>1.158600</td>
      <td>1.368176</td>
      <td>0.264800</td>
      <td>0.541100</td>
      <td>0.233100</td>
      <td>0.138200</td>
      <td>0.223900</td>
      <td>0.498700</td>
      <td>0.272300</td>
      <td>0.407400</td>
      <td>0.434400</td>
      <td>0.


key: _node_content
string_value:
{"id_": "8a42d618-dd15-4e93-a637-4ee372a7bb08", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0cc62b0a-741b-42f4-9780-7cf5deb4ba96", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "a42eb577bc6d183070b970ab98fa77e015c695c7921dbcf8a3b59d47f2ba8f03", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f36e50f-f559-4088-81ef-46a9a65ea10a", "node_type": "1", "metadata": {}, "hash": "108fb5b15066664482823b60bab61e4ec9e696ddca62cf0ada2baeb41f2d74d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38516, "end_char_idx": 40222, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
368296</td>
      <td>0.262800</td>
      <td>0.542400</td>
      <td>0.236400</td>
      <td>0.137400</td>
      <td>0.228100</td>
      <td>0.498500</td>
      <td>0.266500</td>
      <td>0.409000</td>
      <td>0.433000</td>
      <td>0.239900</td>
      <td>0.418500</td>
      <td>0.697500</td>
      <td>0.520500</td>
      <td>0.641000</td>
      <td>0.257500</td>
      <td>0.455700</td>
      <td>0.162600</td>
      <td>0.334800</td>
      <td>0.140200</td>
      <td>0.353800</td>
      <td>0.233200</td>
      <td>0.379600</td>
    </tr>
    <tr>
      <td>23</td>
      <td>1.158600</td>
      <td>1.368176</td>
      <td>0.264800</td>
      <td>0.541100</td>
      <td>0.233100</td>
      <td>0.138200</td>
      <td>0.223900</td>
      <td>0.498700</td>
      <td>0.272300</td>
      <td>0.407400</td>
      <td>0.434400</td>
      <td>0.233100</td>
      <td>0.418300</td>
      <td>0.702000</td>
      <td>0.524400</td>
      <td>0.642300</td>
      <td>0.262300</td>
      <td>0.444300</td>
      <td>0.159700</td>
      <td>0.335300</td>
      <td>0.140500</td>
      <td>0.366200</td>
      <td>0.236900</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>24</td>
      <td>1.049700</td>
      <td>1.355271</td>
      <td>0.269700</td>
      <td>0.549200</td>
      <td>0.239100</td>
      <td>0.134700</td>
      <td>0.229900</td>
      <td>0.519200</td>
      <td>0.274800</td>
      <td>0.412700</td>
      <td>0.437600</td>
      <td>0.245400</td>
      <td>0.417200</td>
      <td>0.711200</td>
      <td>0.523200</td>
      <td>0.644100</td>
      <td>0.272100</td>
      <td>0.440500</td>
      <td>0.166700</td>
      <td>0.341500</td>
      <td>0.137700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "7f36e50f-f559-4088-81ef-46a9a65ea10a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a42d618-dd15-4e93-a637-4ee372a7bb08", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "be68388591b61172fc6ecc2482a4b4f4c523047752a76d84ad7ad153acb94f97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6deb91b0-b583-4939-b904-bdd996693dd0", "node_type": "1", "metadata": {}, "hash": "1ce097a499811ccd2fac259ac29813066670c0cb078fa4b1ea3c1ae03e3b3481", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39369, "end_char_idx": 41088, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
233100</td>
      <td>0.418300</td>
      <td>0.702000</td>
      <td>0.524400</td>
      <td>0.642300</td>
      <td>0.262300</td>
      <td>0.444300</td>
      <td>0.159700</td>
      <td>0.335300</td>
      <td>0.140500</td>
      <td>0.366200</td>
      <td>0.236900</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>24</td>
      <td>1.049700</td>
      <td>1.355271</td>
      <td>0.269700</td>
      <td>0.549200</td>
      <td>0.239100</td>
      <td>0.134700</td>
      <td>0.229900</td>
      <td>0.519200</td>
      <td>0.274800</td>
      <td>0.412700</td>
      <td>0.437600</td>
      <td>0.245400</td>
      <td>0.417200</td>
      <td>0.711200</td>
      <td>0.523200</td>
      <td>0.644100</td>
      <td>0.272100</td>
      <td>0.440500</td>
      <td>0.166700</td>
      <td>0.341500</td>
      <td>0.137700</td>
      <td>0.373800</td>
      <td>0.249000</td>
      <td>0.388000</td>
    </tr>
    <tr>
      <td>25</td>
      <td>1.049700</td>
      <td>1.355180</td>
      <td>0.272500</td>
      <td>0.547900</td>
      <td>0.243800</td>
      <td>0.149700</td>
      <td>0.229900</td>
      <td>0.523100</td>
      <td>0.272500</td>
      <td>0.415700</td>
      <td>0.442200</td>
      <td>0.256200</td>
      <td>0.420200</td>
      <td>0.705800</td>
      <td>0.523900</td>
      <td>0.639600</td>
      <td>0.271700</td>
      <td>0.451900</td>
      <td>0.166300</td>
      <td>0.346900</td>
      <td>0.153700</td>
      <td>0.383100</td>
      <td>0.247000</td>
      <td>0.389300</td>
    </tr>
    <tr>
      <td>26</td>
      <td>1.049700</td>
      <td>1.349337</td>
      <td>0.275600</td>
      <td>0.556300</td>
      <td>0.246400</td>
      <td>0.146700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "6deb91b0-b583-4939-b904-bdd996693dd0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f36e50f-f559-4088-81ef-46a9a65ea10a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "b0329d896ab3f41a34e08c4b03f53cb615ca454e8be37b4d25933fdf47ba0dd7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b84aa76e-91e9-44ce-887c-bdabc8e0d1f4", "node_type": "1", "metadata": {}, "hash": "0ea5fcc2bf037a653b27fc45f254faee0670dedad78e4a966a3041d471cde9a4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40222, "end_char_idx": 41941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
373800</td>
      <td>0.249000</td>
      <td>0.388000</td>
    </tr>
    <tr>
      <td>25</td>
      <td>1.049700</td>
      <td>1.355180</td>
      <td>0.272500</td>
      <td>0.547900</td>
      <td>0.243800</td>
      <td>0.149700</td>
      <td>0.229900</td>
      <td>0.523100</td>
      <td>0.272500</td>
      <td>0.415700</td>
      <td>0.442200</td>
      <td>0.256200</td>
      <td>0.420200</td>
      <td>0.705800</td>
      <td>0.523900</td>
      <td>0.639600</td>
      <td>0.271700</td>
      <td>0.451900</td>
      <td>0.166300</td>
      <td>0.346900</td>
      <td>0.153700</td>
      <td>0.383100</td>
      <td>0.247000</td>
      <td>0.389300</td>
    </tr>
    <tr>
      <td>26</td>
      <td>1.049700</td>
      <td>1.349337</td>
      <td>0.275600</td>
      <td>0.556300</td>
      <td>0.246400</td>
      <td>0.146700</td>
      <td>0.234800</td>
      <td>0.516300</td>
      <td>0.274200</td>
      <td>0.418300</td>
      <td>0.440900</td>
      <td>0.248700</td>
      <td>0.418900</td>
      <td>0.705800</td>
      <td>0.523200</td>
      <td>0.636500</td>
      <td>0.274700</td>
      <td>0.440500</td>
      <td>0.172400</td>
      <td>0.349100</td>
      <td>0.155600</td>
      <td>0.384600</td>
      <td>0.252300</td>
      <td>0.393800</td>
    </tr>
    <tr>
      <td>27</td>
      <td>1.049700</td>
      <td>1.350782</td>
      <td>0.275200</td>
      <td>0.548700</td>
      <td>0.246800</td>
      <td>0.147300</td>
      <td>0.236400</td>
      <td>0.527200</td>
      <td>0.280100</td>
      <td>0.416200</td>
      <td>0.442600</td>
      <td>0.253400</td>
      <td>0.424000</td>
      <td>0.710300</td>
      <td>0.526600</td>
      <td>0.640100</td>
      <td>0.


key: _node_content
string_value:
{"id_": "b84aa76e-91e9-44ce-887c-bdabc8e0d1f4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6deb91b0-b583-4939-b904-bdd996693dd0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "eea4576c6d6873529cc3bea66ea1778f283e2678101b921b93587b016d3cd8aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3be1cac-cb30-4ddf-9c10-ffebd53de173", "node_type": "1", "metadata": {}, "hash": "b3ce0a443be5ab2466ca0c64fd00fabf3259ec1ecc365afdb2c7639f161401bf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41088, "end_char_idx": 42807, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
234800</td>
      <td>0.516300</td>
      <td>0.274200</td>
      <td>0.418300</td>
      <td>0.440900</td>
      <td>0.248700</td>
      <td>0.418900</td>
      <td>0.705800</td>
      <td>0.523200</td>
      <td>0.636500</td>
      <td>0.274700</td>
      <td>0.440500</td>
      <td>0.172400</td>
      <td>0.349100</td>
      <td>0.155600</td>
      <td>0.384600</td>
      <td>0.252300</td>
      <td>0.393800</td>
    </tr>
    <tr>
      <td>27</td>
      <td>1.049700</td>
      <td>1.350782</td>
      <td>0.275200</td>
      <td>0.548700</td>
      <td>0.246800</td>
      <td>0.147300</td>
      <td>0.236400</td>
      <td>0.527200</td>
      <td>0.280100</td>
      <td>0.416200</td>
      <td>0.442600</td>
      <td>0.253400</td>
      <td>0.424000</td>
      <td>0.710300</td>
      <td>0.526600</td>
      <td>0.640100</td>
      <td>0.273200</td>
      <td>0.445600</td>
      <td>0.167000</td>
      <td>0.346900</td>
      <td>0.160100</td>
      <td>0.387700</td>
      <td>0.249200</td>
      <td>0.392900</td>
    </tr>
    <tr>
      <td>28</td>
      <td>1.049700</td>
      <td>1.346533</td>
      <td>0.277000</td>
      <td>0.552800</td>
      <td>0.252900</td>
      <td>0.147400</td>
      <td>0.240000</td>
      <td>0.527600</td>
      <td>0.280900</td>
      <td>0.420900</td>
      <td>0.444100</td>
      <td>0.255500</td>
      <td>0.424500</td>
      <td>0.711200</td>
      <td>0.530200</td>
      <td>0.646800</td>
      <td>0.277400</td>
      <td>0.441800</td>
      <td>0.170900</td>
      <td>0.346900</td>
      <td>0.156600</td>
      <td>0.389200</td>
      <td>0.249600</td>
      <td>0.396000</td>
    </tr>
    <tr>
      <td>29</td>
      <td>0.993700</td>
      <td>1.


key: _node_content
string_value:
{"id_": "e3be1cac-cb30-4ddf-9c10-ffebd53de173", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b84aa76e-91e9-44ce-887c-bdabc8e0d1f4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "5b33e1d2e1d5d67dba5584bb5d11b6244c552e10c848f8be320d7b97b46991ff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92f15471-ed88-4dc4-930c-f5a8d39c9bbd", "node_type": "1", "metadata": {}, "hash": "59982fcea0cccdb244a5f5733c581b0f3199be6ada77b32787c79b333e2706c8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41941, "end_char_idx": 43660, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


