# embedding_metadata

key: _node_content
string_value:
{"id_": "2fbdf6b0-6533-4233-9410-eb8c94d5d897", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ebe5476-1cb6-465c-9563-a8060530d330", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "b73b366894eb849dd87d3b79c63036701f91ae8e746a90f77d83e1c1e4c4b251", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c601e0b-75e7-4365-9e70-1c2fd0574d67", "node_type": "1", "metadata": {}, "hash": "654a3cc059ce368316d40b0afbe0907eb3c10a56d33a23321eaed3c70be3c3ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24129, "end_char_idx": 27390, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass your `compute_metrics` function to [`KerasMetricCallback`],
and use the [`PushToHubCallback`] to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(
...     metric_fn=compute_metrics, eval_dataset=tf_eval_dataset, batch_size=batch_size, label_cols=["labels"]
... )

>>> push_to_hub_callback = PushToHubCallback(output_dir="scene_segmentation", tokenizer=image_processor)

>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(
...     tf_train_dataset,
...     validation_data=tf_eval_dataset,
...     callbacks=callbacks,
...     epochs=num_epochs,
... )
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>

### Inference

Great, now that you've finetuned a model, you can use it for inference!

Reload the dataset and load an image for inference.

```py
>>> from datasets import load_dataset

>>> ds = load_dataset("scene_parse_150", split="train[:50]")
>>> ds = ds.train_test_split(test_size=0.2)
>>> test_ds = ds["test"]
>>> image = ds["test"][0]["image"]
>>> image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-image.png" alt="Image of bedroom"/>
</div>

<frameworkcontent>
<pt>

We will now see how to infer without a pipeline. Process the image with an image processor and place the `pixel_values` on a GPU:

```py
>>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  # use GPU if available, otherwise use a CPU
>>> encoding = image_processor(image, return_tensors="pt")
>>> pixel_values = encoding.pixel_values.to(device)
```

Pass your input to the model and return the `logits`:

```py
>>> outputs = model(pixel_values=pixel_values)
>>> logits = outputs.logits.cpu()
```

Next, rescale the logits to the original image size:

```py
>>> upsampled_logits = nn.functional.interpolate(
...     logits,
...     size=image.size[::-1],
...     mode="bilinear",
...     align_corners=False,
... )

>>> pred_seg = upsampled_logits.argmax(dim=1)[0]
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
Load an image processor to preprocess the image and return the input as TensorFlow tensors:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/scene_segmentation")
>>> inputs = image_processor(image, return_tensors="tf")
```

Pass your input to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("MariaK/scene_segmentation")
>>> logits = model(**inputs).logits
```

Next, rescale the logits to the original image size and apply argmax on the class dimension:
```py
>>> logits = tf.transpose(logits, [0, 2, 3, 1])

>>> upsampled_logits = tf.image.resize(
...     logits,
...     # We reverse the shape of `image` because `image.size` returns width and height.


key: _node_content
string_value:
{"id_": "4c601e0b-75e7-4365-9e70-1c2fd0574d67", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fbdf6b0-6533-4233-9410-eb8c94d5d897", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "6705a654985aff0026a211230132fdc748344f16d1015f1a609cf4b3b0a595f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad051214-04be-4317-9332-2f475c8b15a1", "node_type": "1", "metadata": {}, "hash": "c76b62c8021d262578030a4d6397245ab30aeafd02a7d38df60b9c3d9e7f4d1c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25750, "end_char_idx": 28315, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Process the image with an image processor and place the `pixel_values` on a GPU:

```py
>>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")  # use GPU if available, otherwise use a CPU
>>> encoding = image_processor(image, return_tensors="pt")
>>> pixel_values = encoding.pixel_values.to(device)
```

Pass your input to the model and return the `logits`:

```py
>>> outputs = model(pixel_values=pixel_values)
>>> logits = outputs.logits.cpu()
```

Next, rescale the logits to the original image size:

```py
>>> upsampled_logits = nn.functional.interpolate(
...     logits,
...     size=image.size[::-1],
...     mode="bilinear",
...     align_corners=False,
... )

>>> pred_seg = upsampled_logits.argmax(dim=1)[0]
```

</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
Load an image processor to preprocess the image and return the input as TensorFlow tensors:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/scene_segmentation")
>>> inputs = image_processor(image, return_tensors="tf")
```

Pass your input to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSemanticSegmentation

>>> model = TFAutoModelForSemanticSegmentation.from_pretrained("MariaK/scene_segmentation")
>>> logits = model(**inputs).logits
```

Next, rescale the logits to the original image size and apply argmax on the class dimension:
```py
>>> logits = tf.transpose(logits, [0, 2, 3, 1])

>>> upsampled_logits = tf.image.resize(
...     logits,
...     # We reverse the shape of `image` because `image.size` returns width and height.
...     image.size[::-1],
... )

>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]
```

</tf>
</frameworkcontent>

To visualize the results, load the [dataset color palette](https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51) as `ade_palette()` that maps each class to their RGB values.

```py
def ade_palette():
  return np.asarray([
      [0, 0, 0],
      [120, 120, 120],
      [180, 120, 120],
      [6, 230, 230],
      [80, 50, 50],
      [4, 200, 3],
      [120, 120, 80],
      [140, 140, 140],
      [204, 5, 255],
      [230, 230, 230],
      [4, 250, 7],
      [224, 5, 255],
      [235, 255, 7],
      [150, 5, 61],
      [120, 120, 70],
      [8, 255, 51],
      [255, 6, 82],
      [143, 255, 140],
      [204, 255, 4],
      [255, 51, 7],
      [204, 70, 3],
      [0, 102, 200],
      [61, 230, 250],
      [255, 6,


key: _node_content
string_value:
{"id_": "ad051214-04be-4317-9332-2f475c8b15a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4c601e0b-75e7-4365-9e70-1c2fd0574d67", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "0e0aa44835499848ea21bf0f138260a05a7d20c47b2ab482e78bea0064cc849c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f231b7d8-b1b4-4d11-8c66-04990df78a4c", "node_type": "1", "metadata": {}, "hash": "8a2243df478a5ee3610ff7bb2c6dfe62393290c9fe20d73c944b8f4849da0583", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27391, "end_char_idx": 29063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     image.size[::-1],
... )

>>> pred_seg = tf.math.argmax(upsampled_logits, axis=-1)[0]
```

</tf>
</frameworkcontent>

To visualize the results, load the [dataset color palette](https://github.com/tensorflow/models/blob/3f1ca33afe3c1631b733ea7e40c294273b9e406d/research/deeplab/utils/get_dataset_colormap.py#L51) as `ade_palette()` that maps each class to their RGB values.

```py
def ade_palette():
  return np.asarray([
      [0, 0, 0],
      [120, 120, 120],
      [180, 120, 120],
      [6, 230, 230],
      [80, 50, 50],
      [4, 200, 3],
      [120, 120, 80],
      [140, 140, 140],
      [204, 5, 255],
      [230, 230, 230],
      [4, 250, 7],
      [224, 5, 255],
      [235, 255, 7],
      [150, 5, 61],
      [120, 120, 70],
      [8, 255, 51],
      [255, 6, 82],
      [143, 255, 140],
      [204, 255, 4],
      [255, 51, 7],
      [204, 70, 3],
      [0, 102, 200],
      [61, 230, 250],
      [255, 6, 51],
      [11, 102, 255],
      [255, 7, 71],
      [255, 9, 224],
      [9, 7, 230],
      [220, 220, 220],
      [255, 9, 92],
      [112, 9, 255],
      [8, 255, 214],
      [7, 255, 224],
      [255, 184, 6],
      [10, 255, 71],
      [255, 41, 10],
      [7, 255, 255],
      [224, 255, 8],
      [102, 8, 255],
      [255, 61, 6],
      [255, 194, 7],
      [255, 122, 8],
      [0, 255, 20],
      [255, 8, 41],
      [255, 5, 153],
      [6, 51, 255],
      [235, 12, 255],
      [160, 150, 20],
      [0, 163, 255],
      [140, 140, 140],
      [250, 10, 15],
      [20, 255, 0],
      [31, 255, 0],
      [255, 31, 0],
      [255, 224, 0],
      [153, 255, 0],
      [0, 0, 255],
      [255, 71, 0],
      [0, 235, 255],
      [0, 173,


key: _node_content
string_value:
{"id_": "f231b7d8-b1b4-4d11-8c66-04990df78a4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad051214-04be-4317-9332-2f475c8b15a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "1f587021e32c4cc67eddfa47f6b89b483d4f79ecb5af1c32db302f15483e7053", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a91a501d-c388-48e0-82ba-94587e273ac5", "node_type": "1", "metadata": {}, "hash": "65527d728b9d77eff1556d1fdd0bdec48b564fcdf2bf2e970072adae620c89b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28313, "end_char_idx": 29781, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
6, 51],
      [11, 102, 255],
      [255, 7, 71],
      [255, 9, 224],
      [9, 7, 230],
      [220, 220, 220],
      [255, 9, 92],
      [112, 9, 255],
      [8, 255, 214],
      [7, 255, 224],
      [255, 184, 6],
      [10, 255, 71],
      [255, 41, 10],
      [7, 255, 255],
      [224, 255, 8],
      [102, 8, 255],
      [255, 61, 6],
      [255, 194, 7],
      [255, 122, 8],
      [0, 255, 20],
      [255, 8, 41],
      [255, 5, 153],
      [6, 51, 255],
      [235, 12, 255],
      [160, 150, 20],
      [0, 163, 255],
      [140, 140, 140],
      [250, 10, 15],
      [20, 255, 0],
      [31, 255, 0],
      [255, 31, 0],
      [255, 224, 0],
      [153, 255, 0],
      [0, 0, 255],
      [255, 71, 0],
      [0, 235, 255],
      [0, 173, 255],
      [31, 0, 255],
      [11, 200, 200],
      [255, 82, 0],
      [0, 255, 245],
      [0, 61, 255],
      [0, 255, 112],
      [0, 255, 133],
      [255, 0, 0],
      [255, 163, 0],
      [255, 102, 0],
      [194, 255, 0],
      [0, 143, 255],
      [51, 255, 0],
      [0, 82, 255],
      [0, 255, 41],
      [0, 255, 173],
      [10, 0, 255],
      [173, 255, 0],
      [0, 255, 153],
      [255, 92, 0],
      [255, 0, 255],
      [255, 0, 245],
      [255, 0, 102],
      [255, 173, 0],
      [255, 0, 20],
      [255, 184, 184],
      [0, 31, 255],
      [0, 255, 61],
      [0, 71, 255],
      [255, 0, 204],
      [0, 255, 194],
      [0, 255, 82],
      [0, 10, 255],
      [0, 112, 255],
      [51,


key: _node_content
string_value:
{"id_": "a91a501d-c388-48e0-82ba-94587e273ac5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f231b7d8-b1b4-4d11-8c66-04990df78a4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "23b3dc3f22245c634c4c1e302d8242d7651db0b93e0c0cd7c3d162e9a77a3877", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30b4578d-877a-4577-8d3b-d57c68d8c696", "node_type": "1", "metadata": {}, "hash": "a1e219b327f2f232ebb37965203f0aa2e13799f883996ce398724fc32606ed52", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29038, "end_char_idx": 30512, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
235, 255],
      [0, 173, 255],
      [31, 0, 255],
      [11, 200, 200],
      [255, 82, 0],
      [0, 255, 245],
      [0, 61, 255],
      [0, 255, 112],
      [0, 255, 133],
      [255, 0, 0],
      [255, 163, 0],
      [255, 102, 0],
      [194, 255, 0],
      [0, 143, 255],
      [51, 255, 0],
      [0, 82, 255],
      [0, 255, 41],
      [0, 255, 173],
      [10, 0, 255],
      [173, 255, 0],
      [0, 255, 153],
      [255, 92, 0],
      [255, 0, 255],
      [255, 0, 245],
      [255, 0, 102],
      [255, 173, 0],
      [255, 0, 20],
      [255, 184, 184],
      [0, 31, 255],
      [0, 255, 61],
      [0, 71, 255],
      [255, 0, 204],
      [0, 255, 194],
      [0, 255, 82],
      [0, 10, 255],
      [0, 112, 255],
      [51, 0, 255],
      [0, 194, 255],
      [0, 122, 255],
      [0, 255, 163],
      [255, 153, 0],
      [0, 255, 10],
      [255, 112, 0],
      [143, 255, 0],
      [82, 0, 255],
      [163, 255, 0],
      [255, 235, 0],
      [8, 184, 170],
      [133, 0, 255],
      [0, 255, 92],
      [184, 0, 255],
      [255, 0, 31],
      [0, 184, 255],
      [0, 214, 255],
      [255, 0, 112],
      [92, 255, 0],
      [0, 224, 255],
      [112, 224, 255],
      [70, 184, 160],
      [163, 0, 255],
      [153, 0, 255],
      [71, 255, 0],
      [255, 0, 163],
      [255, 204, 0],
      [255, 0, 143],
      [0, 255, 235],
      [133, 255, 0],
      [255, 0, 235],
      [245, 0, 255],
      [255, 0, 122],
      [255, 245, 0],
      [10,


key: _node_content
string_value:
{"id_": "30b4578d-877a-4577-8d3b-d57c68d8c696", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a91a501d-c388-48e0-82ba-94587e273ac5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "5d46c141dc355b7a2c3a96f1d2f1a2cb1c63d5f00fb7ecbede2efb7d5f14aa04", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80b8870e-eb08-491a-b3ff-7f7cb3b91cf9", "node_type": "1", "metadata": {}, "hash": "8820c311ab122c4c017562b695f18cc3f2cf85f41d7f71e754c9f88f6b5c3036", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
112, 255],
      [51, 0, 255],
      [0, 194, 255],
      [0, 122, 255],
      [0, 255, 163],
      [255, 153, 0],
      [0, 255, 10],
      [255, 112, 0],
      [143, 255, 0],
      [82, 0, 255],
      [163, 255, 0],
      [255, 235, 0],
      [8, 184, 170],
      [133, 0, 255],
      [0, 255, 92],
      [184, 0, 255],
      [255, 0, 31],
      [0, 184, 255],
      [0, 214, 255],
      [255, 0, 112],
      [92, 255, 0],
      [0, 224, 255],
      [112, 224, 255],
      [70, 184, 160],
      [163, 0, 255],
      [153, 0, 255],
      [71, 255, 0],
      [255, 0, 163],
      [255, 204, 0],
      [255, 0, 143],
      [0, 255, 235],
      [133, 255, 0],
      [255, 0, 235],
      [245, 0, 255],
      [255, 0, 122],
      [255, 245, 0],
      [10, 190, 212],
      [214, 255, 0],
      [0, 204, 255],
      [20, 0, 255],
      [255, 255, 0],
      [0, 153, 255],
      [0, 41, 255],
      [0, 255, 204],
      [41, 0, 255],
      [41, 255, 0],
      [173, 0, 255],
      [0, 245, 255],
      [71, 0, 255],
      [122, 0, 255],
      [0, 255, 184],
      [0, 92, 255],
      [184, 255, 0],
      [0, 133, 255],
      [255, 214, 0],
      [25, 194, 194],
      [102, 255, 0],
      [92, 0, 255],
  ])
```

Then you can combine and plot your image and the predicted segmentation map:

```py
>>> import matplotlib.pyplot as plt
>>> import numpy as np

>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)
>>> palette = np.array(ade_palette())
>>> for label, color in enumerate(palette):
.     color_seg[pred_seg == label, :] = color
>>> color_seg = color_seg[. ::-1]  # convert to BGR

>>> img = np.array(image) * 0.5 + color_seg * 0.


key: _node_content
string_value:
{"id_": "80b8870e-eb08-491a-b3ff-7f7cb3b91cf9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3557c9ca-fe71-4e83-b266-b8fdc87d4f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "088380408e5d8d3e8635cbee00fc083f61fe82c70ae8ed27b60d472a6bd0b6b1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30b4578d-877a-4577-8d3b-d57c68d8c696", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/semantic_segmentation/", "title": "Image Segmentation", "tokens": 8705, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "d5b6a6b4d3d8d6861b8e42c69bff18dfefbd65fca78d52d5456551d3e8f44b78", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0, 122],
      [255, 245, 0],
      [10, 190, 212],
      [214, 255, 0],
      [0, 204, 255],
      [20, 0, 255],
      [255, 255, 0],
      [0, 153, 255],
      [0, 41, 255],
      [0, 255, 204],
      [41, 0, 255],
      [41, 255, 0],
      [173, 0, 255],
      [0, 245, 255],
      [71, 0, 255],
      [122, 0, 255],
      [0, 255, 184],
      [0, 92, 255],
      [184, 255, 0],
      [0, 133, 255],
      [255, 214, 0],
      [25, 194, 194],
      [102, 255, 0],
      [92, 0, 255],
  ])
```

Then you can combine and plot your image and the predicted segmentation map:

```py
>>> import matplotlib.pyplot as plt
>>> import numpy as np

>>> color_seg = np.zeros((pred_seg.shape[0], pred_seg.shape[1], 3), dtype=np.uint8)
>>> palette = np.array(ade_palette())
>>> for label, color in enumerate(palette):
.     color_seg[pred_seg == label, :] = color
>>> color_seg = color_seg[. ::-1]  # convert to BGR

>>> img = np.array(image) * 0.5 + color_seg * 0.5  # plot the image with the segmentation map
>>> img = img.astype(np.uint8)

>>> plt.figure(figsize=(15, 10))
>>> plt.imshow(img)
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/semantic-seg-preds.png" alt="Image of bedroom overlaid with segmentation map"/>
</div>


key: _node_content
string_value:
{"id_": "e6cc8f8e-cde1-4b3c-bea3-e7a95f2fb1e0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1da74154-1685-4312-917a-3fbb3cc19190", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "caa3a736129b37568c0b7b754b4c125c6a578662680a9aabdd146105cda426de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "413edc33-5e1d-4d9e-9ede-199b2cc35721", "node_type": "1", "metadata": {}, "hash": "c469c64c686b38b6063bb340fd071e6777e3cc9aa72c916bf687c08dc232085d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image captioning

[[open-in-colab]]

Image captioning is the task of predicting a caption for a given image. Common real world applications of it include
aiding visually impaired people that can help them navigate through different situations. Therefore, image captioning
helps to improve content accessibility for people by describing images to them.

This guide will show you how to:

* Fine-tune an image captioning model.
* Use the fine-tuned model for inference. 

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate -q
pip install jiwer -q
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:


```python
from huggingface_hub import notebook_login

notebook_login()
```

## Load the Pokémon BLIP captions dataset

Use the 🤗 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset
in PyTorch, you can follow [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GIT/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb). 


```python
from datasets import load_dataset

ds = load_dataset("lambdalabs/pokemon-blip-captions")
ds
```
```bash
DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 833
    })
})
```

The dataset has two features, `image` and `text`.

<Tip>

Many image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training. 

</Tip>

Split the dataset’s train split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:


```python
ds = ds["train"].train_test_split(test_size=0.1)
train_ds = ds["train"]
test_ds = ds["test"]
```

Let's visualize a couple of samples from the training set. 


```python
from textwrap import wrap
import matplotlib.pyplot as plt
import numpy as np


def plot_images(images, captions):
    plt.figure(figsize=(20, 20))
    for i in range(len(images)):
        ax = plt.subplot(1, len(images), i + 1)
        caption = captions[i]
        caption = "\n".join(wrap(caption, 12))
        plt.title(caption)
        plt.imshow(images[i])
        plt.axis("off")


key: _node_content
string_value:
{"id_": "413edc33-5e1d-4d9e-9ede-199b2cc35721", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1da74154-1685-4312-917a-3fbb3cc19190", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "caa3a736129b37568c0b7b754b4c125c6a578662680a9aabdd146105cda426de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e6cc8f8e-cde1-4b3c-bea3-e7a95f2fb1e0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7f2e2dda8d15f7e29a692a072780894cc630fc6c1bbcf7d36932bd4e7a55edc4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c987c06-9b99-4e6a-9052-febeaa33d287", "node_type": "1", "metadata": {}, "hash": "f49ccd7582a09ead0a203446ff9618ecaa9be430bd6807e6068076d0b6a013ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 789, "end_char_idx": 3953, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from huggingface_hub import notebook_login

notebook_login()
```

## Load the Pokémon BLIP captions dataset

Use the 🤗 Dataset library to load a dataset that consists of {image-caption} pairs. To create your own image captioning dataset
in PyTorch, you can follow [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/GIT/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb). 


```python
from datasets import load_dataset

ds = load_dataset("lambdalabs/pokemon-blip-captions")
ds
```
```bash
DatasetDict({
    train: Dataset({
        features: ['image', 'text'],
        num_rows: 833
    })
})
```

The dataset has two features, `image` and `text`.

<Tip>

Many image captioning datasets contain multiple captions per image. In those cases, a common strategy is to randomly sample a caption amongst the available ones during training. 

</Tip>

Split the dataset’s train split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:


```python
ds = ds["train"].train_test_split(test_size=0.1)
train_ds = ds["train"]
test_ds = ds["test"]
```

Let's visualize a couple of samples from the training set. 


```python
from textwrap import wrap
import matplotlib.pyplot as plt
import numpy as np


def plot_images(images, captions):
    plt.figure(figsize=(20, 20))
    for i in range(len(images)):
        ax = plt.subplot(1, len(images), i + 1)
        caption = captions[i]
        caption = "\n".join(wrap(caption, 12))
        plt.title(caption)
        plt.imshow(images[i])
        plt.axis("off")


sample_images_to_visualize = [np.array(train_ds[i]["image"]) for i in range(5)]
sample_captions = [train_ds[i]["text"] for i in range(5)]
plot_images(sample_images_to_visualize, sample_captions)
```
    
<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_training_images_image_cap.png" alt="Sample training images"/>
</div>

## Preprocess the dataset

Since the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.

To do so, load the processor class associated with the model you are about to fine-tune. 

```python
from transformers import AutoProcessor

checkpoint = "microsoft/git-base"
processor = AutoProcessor.from_pretrained(checkpoint)
```

The processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption. 

```python
def transforms(example_batch):
    images = [x for x in example_batch["image"]]
    captions = [x for x in example_batch["text"]]
    inputs = processor(images=images, text=captions, padding="max_length")
    inputs.update({"labels": inputs["input_ids"]})
    return inputs


train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
```

With the dataset ready, you can now set up the model for fine-tuning. 

## Load a base model

Load the ["microsoft/git-base"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.


key: _node_content
string_value:
{"id_": "0c987c06-9b99-4e6a-9052-febeaa33d287", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1da74154-1685-4312-917a-3fbb3cc19190", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "caa3a736129b37568c0b7b754b4c125c6a578662680a9aabdd146105cda426de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "413edc33-5e1d-4d9e-9ede-199b2cc35721", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9538932239eedc3a7deedf07d7d6c12b46e2d3f03e0c8833d5661275a5e649a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b76f4978-be7f-42d9-8812-26232a7c4043", "node_type": "1", "metadata": {}, "hash": "1f1680a6ba28c4f5e499e7141a37120323fd0a735223608b73bd21b35475c047", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2357, "end_char_idx": 5146, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
sample_images_to_visualize = [np.array(train_ds[i]["image"]) for i in range(5)]
sample_captions = [train_ds[i]["text"] for i in range(5)]
plot_images(sample_images_to_visualize, sample_captions)
```
    
<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_training_images_image_cap.png" alt="Sample training images"/>
</div>

## Preprocess the dataset

Since the dataset has two modalities (image and text), the pre-processing pipeline will preprocess images and the captions.

To do so, load the processor class associated with the model you are about to fine-tune. 

```python
from transformers import AutoProcessor

checkpoint = "microsoft/git-base"
processor = AutoProcessor.from_pretrained(checkpoint)
```

The processor will internally pre-process the image (which includes resizing, and pixel scaling) and tokenize the caption. 

```python
def transforms(example_batch):
    images = [x for x in example_batch["image"]]
    captions = [x for x in example_batch["text"]]
    inputs = processor(images=images, text=captions, padding="max_length")
    inputs.update({"labels": inputs["input_ids"]})
    return inputs


train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
```

With the dataset ready, you can now set up the model for fine-tuning. 

## Load a base model

Load the ["microsoft/git-base"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.


```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(checkpoint)
```

## Evaluate

Image captioning models are typically evaluated with the [Rouge Score](https://huggingface.co/spaces/evaluate-metric/rouge) or [Word Error Rate](https://huggingface.co/spaces/evaluate-metric/wer). For this guide, you will use the Word Error Rate (WER). 

We use the 🤗 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to [this guide](https://huggingface.co/spaces/evaluate-metric/wer). 


```python
from evaluate import load
import torch

wer = load("wer")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predicted = logits.argmax(-1)
    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)
    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)
    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)
    return {"wer_score": wer_score}
```

## Train!

Now, you are ready to start fine-tuning the model. You will use the 🤗 [`Trainer`] for this. 

First, define the training arguments using [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "b76f4978-be7f-42d9-8812-26232a7c4043", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1da74154-1685-4312-917a-3fbb3cc19190", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "caa3a736129b37568c0b7b754b4c125c6a578662680a9aabdd146105cda426de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c987c06-9b99-4e6a-9052-febeaa33d287", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c0d08b63c18a0e1c7cbbf090b87881b88bbbf6738dd70c1ecbbe391b04e2475e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd985d95-9c89-4bdd-80a8-453c69105df1", "node_type": "1", "metadata": {}, "hash": "d47d08787f0017b2e6b4f1b639a84997b6effc52688873af0686e1c9e9071aed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3580, "end_char_idx": 6411, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
train_ds.set_transform(transforms)
test_ds.set_transform(transforms)
```

With the dataset ready, you can now set up the model for fine-tuning. 

## Load a base model

Load the ["microsoft/git-base"](https://huggingface.co/microsoft/git-base) into a [`AutoModelForCausalLM`](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) object.


```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(checkpoint)
```

## Evaluate

Image captioning models are typically evaluated with the [Rouge Score](https://huggingface.co/spaces/evaluate-metric/rouge) or [Word Error Rate](https://huggingface.co/spaces/evaluate-metric/wer). For this guide, you will use the Word Error Rate (WER). 

We use the 🤗 Evaluate library to do so. For potential limitations and other gotchas of the WER, refer to [this guide](https://huggingface.co/spaces/evaluate-metric/wer). 


```python
from evaluate import load
import torch

wer = load("wer")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predicted = logits.argmax(-1)
    decoded_labels = processor.batch_decode(labels, skip_special_tokens=True)
    decoded_predictions = processor.batch_decode(predicted, skip_special_tokens=True)
    wer_score = wer.compute(predictions=decoded_predictions, references=decoded_labels)
    return {"wer_score": wer_score}
```

## Train!

Now, you are ready to start fine-tuning the model. You will use the 🤗 [`Trainer`] for this. 

First, define the training arguments using [`TrainingArguments`].


```python
from transformers import TrainingArguments, Trainer

model_name = checkpoint.split("/")[1]

training_args = TrainingArguments(
    output_dir=f"{model_name}-pokemon",
    learning_rate=5e-5,
    num_train_epochs=50,
    fp16=True,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    gradient_accumulation_steps=2,
    save_total_limit=3,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    logging_steps=50,
    remove_unused_columns=False,
    push_to_hub=True,
    label_names=["labels"],
    load_best_model_at_end=True,
)
```

Then pass them along with the datasets and the model to 🤗 Trainer. 

```python
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    compute_metrics=compute_metrics,
)
```

To start training, simply call [`~Trainer.train`] on the [`Trainer`] object.

```python 
trainer.train()
```

You should see the training loss drop smoothly as training progresses.

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method so everyone can use your model:


```python
trainer.push_to_hub()
```

## Inference

Take a sample image from `test_ds` to test the model.


key: _node_content
string_value:
{"id_": "dd985d95-9c89-4bdd-80a8-453c69105df1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1da74154-1685-4312-917a-3fbb3cc19190", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "caa3a736129b37568c0b7b754b4c125c6a578662680a9aabdd146105cda426de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b76f4978-be7f-42d9-8812-26232a7c4043", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_captioning/", "title": "Image captioning", "tokens": 1900, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "55cc2386651fc5a9761fbea0dbffff10b1ec826d8dc46de37baffc3dbe728962", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5149, "end_char_idx": 7385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import TrainingArguments, Trainer

model_name = checkpoint.split("/")[1]

training_args = TrainingArguments(
    output_dir=f"{model_name}-pokemon",
    learning_rate=5e-5,
    num_train_epochs=50,
    fp16=True,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    gradient_accumulation_steps=2,
    save_total_limit=3,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
    logging_steps=50,
    remove_unused_columns=False,
    push_to_hub=True,
    label_names=["labels"],
    load_best_model_at_end=True,
)
```

Then pass them along with the datasets and the model to 🤗 Trainer. 

```python
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    compute_metrics=compute_metrics,
)
```

To start training, simply call [`~Trainer.train`] on the [`Trainer`] object.

```python 
trainer.train()
```

You should see the training loss drop smoothly as training progresses.

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method so everyone can use your model:


```python
trainer.push_to_hub()
```

## Inference

Take a sample image from `test_ds` to test the model.


```python
from PIL import Image
import requests

url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/pokemon.png"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/test_image_image_cap.png" alt="Test image"/>
</div>
    
Prepare image for the model.

```python
device = "cuda" if torch.cuda.is_available() else "cpu"

inputs = processor(images=image, return_tensors="pt").to(device)
pixel_values = inputs.pixel_values
```

Call [`generate`] and decode the predictions. 

```python
generated_ids = model.generate(pixel_values=pixel_values, max_length=50)
generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_caption)
```
```bash
a drawing of a pink and blue pokemon
```

Looks like the fine-tuned model generated a pretty good caption!


key: _node_content
string_value:
{"id_": "3b5cd974-4f94-4cc3-8eb5-cdbe356a424a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8cdc3645-273c-4fdb-811c-f59cb6fe7dfd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5441f1c819aab9a3c5006b56c0f63c5b5a0cd54b0d44315bc44d9a29bb9197d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "64e71226-b23e-4175-a92f-6dadde6f2e17", "node_type": "1", "metadata": {}, "hash": "dfff0139cabab5dd294e0f21af1aacc0a2872d281258f9ba9804f68f50435191", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2561, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image-text-to-text

[[open-in-colab]]

Image-text-to-text models, also known as vision language models (VLMs), are language models that take an image input. These models can tackle various tasks, from visual question answering to image segmentation. This task shares many similarities with image-to-text, but with some overlapping use cases like image captioning. Image-to-text models only take image inputs and often accomplish a specific task, whereas VLMs take open-ended text and image inputs and are more generalist models.

In this guide, we provide a brief overview of VLMs and show how to use them with Transformers for inference.

To begin with, there are multiple types of VLMs:
- base models used for fine-tuning
- chat fine-tuned models for conversation
- instruction fine-tuned models

This guide focuses on inference with an instruction-tuned model. 

Let's begin installing the dependencies.

```bash
pip install -q transformers accelerate flash_attn 
```

Let's initialize the model and the processor. 

```python
from transformers import AutoProcessor, Idefics2ForConditionalGeneration
import torch

device = torch.device("cuda")
model = Idefics2ForConditionalGeneration.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to(device)

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
```

This model has a [chat template](./chat_templating) that helps user parse chat outputs. Moreover, the model can also accept multiple images as input in a single conversation or message. We will now prepare the inputs. 

The image inputs look like the following.

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png" alt="Two cats sitting on a net"/>
</div>

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="A bee on a pink flower"/>
</div>


```python
from PIL import Image
import requests

img_urls =["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
           "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"]
images = [Image.open(requests.get(img_urls[0], stream=True).raw),
          Image.open(requests.get(img_urls[1], stream=True).raw)]
```

Below is an example of the chat template. We can feed conversation turns and the last message as an input by appending it at the end of the template.


key: _node_content
string_value:
{"id_": "64e71226-b23e-4175-a92f-6dadde6f2e17", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8cdc3645-273c-4fdb-811c-f59cb6fe7dfd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5441f1c819aab9a3c5006b56c0f63c5b5a0cd54b0d44315bc44d9a29bb9197d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b5cd974-4f94-4cc3-8eb5-cdbe356a424a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b83c6b6163a318b1724859dffd811a830042a4436f62dee275f1cb3ba62636d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad518d9b-119e-4029-8089-b78d71c8ed0f", "node_type": "1", "metadata": {}, "hash": "fbd0a5c8b592fadba8b90674eb436326c6b9d909a205e7c764876ad2e3e6cc31", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2025, "end_char_idx": 4500, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from PIL import Image
import requests

img_urls =["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
           "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"]
images = [Image.open(requests.get(img_urls[0], stream=True).raw),
          Image.open(requests.get(img_urls[1], stream=True).raw)]
```

Below is an example of the chat template. We can feed conversation turns and the last message as an input by appending it at the end of the template. 


```python
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image we can see two cats on the nets."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },       
]
```

We will now call the processors' [`~ProcessorMixin.apply_chat_template`] method to preprocess its output along with the image inputs.

```python
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[images[0], images[1]], return_tensors="pt").to(device)
```

We can now pass the preprocessed inputs to the model.

```python
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)
## ['User: What do we see in this image? \nAssistant: In this image we can see two cats on the nets. \nUser: And how about this image? \nAssistant: In this image we can see flowers, plants and insect.']
```

## Streaming

We can use [text streaming](./generation_strategies#streaming) for a better generation experience. Transformers supports streaming with the [`TextStreamer`] or [`TextIteratorStreamer`] classes. We will use the [`TextIteratorStreamer`] with IDEFICS-8B.

Assume we have an application that keeps chat history and takes in the new user input. We will preprocess the inputs as usual and initialize [`TextIteratorStreamer`] to handle the generation in a separate thread. This allows you to stream the generated text tokens in real-time. Any generation arguments can be passed to [`TextIteratorStreamer`].


key: _node_content
string_value:
{"id_": "ad518d9b-119e-4029-8089-b78d71c8ed0f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8cdc3645-273c-4fdb-811c-f59cb6fe7dfd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5441f1c819aab9a3c5006b56c0f63c5b5a0cd54b0d44315bc44d9a29bb9197d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "64e71226-b23e-4175-a92f-6dadde6f2e17", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_text_to_text/", "title": "Image-text-to-text", "tokens": 1936, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19dc99d2126c21cca0c4efd9c14c61c415c3bc6a34d6586665d2de50dfd28107", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4503, "end_char_idx": 7704, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import time
from transformers import TextIteratorStreamer
from threading import Thread

def model_inference(
    user_prompt,
    chat_history,
    max_new_tokens,
    images
):
    user_prompt = {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": user_prompt},
        ]
    }
    chat_history.append(user_prompt)
    streamer = TextIteratorStreamer(
        processor.tokenizer,
        skip_prompt=True,
        timeout=5.0,
    )

    generation_args = {
        "max_new_tokens": max_new_tokens,
        "streamer": streamer,
        "do_sample": False
    }

    # add_generation_prompt=True makes model generate bot response
    prompt = processor.apply_chat_template(chat_history, add_generation_prompt=True)
    inputs = processor(
        text=prompt,
        images=images,
        return_tensors="pt",
    ).to(device)
    generation_args.update(inputs)

    thread = Thread(
        target=model.generate,
        kwargs=generation_args,
    )
    thread.start()

    acc_text = ""
    for text_token in streamer:
        time.sleep(0.04)
        acc_text += text_token
        if acc_text.endswith("<end_of_utterance>"):
            acc_text = acc_text[:-18]
        yield acc_text
    
    thread.join()
```

Now let's call the `model_inference` function we created and stream the values. 

```python
generator = model_inference(
    user_prompt="And what is in this image?",
    chat_history=messages,
    max_new_tokens=100,
    images=images
)

for value in generator:
  print(value)

# In
# In this
# In this image ...
```

## Fit models in smaller hardware

VLMs are often large and need to be optimized to fit in smaller hardware. Transformers supports many model quantization libraries, and here we will only show int8 quantization with [Quanto](./quantization/quanto#quanto). int8 quantization offers memory improvements up to 75 percent (if all weights are quantized). However it is no free lunch, since 8-bit is not a CUDA-native precision, the weights are quantized back and forth on the fly, which adds up to latency. 

First, install dependencies.

```bash
pip install -U quanto bitsandbytes
```

To quantize a model during loading, we need to first create [`QuantoConfig`]. Then load the model as usual, but pass `quantization_config` during model initialization.

```python
from transformers import Idefics2ForConditionalGeneration, AutoTokenizer, QuantoConfig

model_id = "HuggingFaceM4/idefics2-8b"
quantization_config = QuantoConfig(weights="int8")
quantized_model = Idefics2ForConditionalGeneration.from_pretrained(model_id, device_map="cuda", quantization_config=quantization_config)
```

And that's it, we can use the model the same way with no changes. 

## Further Reading

Here are some more resources for the image-text-to-text task.

- [Image-text-to-text task page](https://huggingface.co/tasks/image-text-to-text) covers model types, use cases, datasets, and more. 
- [Vision Language Models Explained](https://huggingface.co/blog/vlms) is a blog post that covers everything about vision language models and supervised fine-tuning using [TRL](https://huggingface.co/docs/trl/en/index).


key: _node_content
string_value:
{"id_": "c70a1d54-d554-480c-a58e-0e80e535de01", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "33619358-f15e-4b6b-9690-e5d07dd43871", "node_type": "1", "metadata": {}, "hash": "6a5369e3d9198abbec82957c2ef70bb255363843ea67c8b218315278721eb334", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2401, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Object detection

[[open-in-colab]]

Object detection is the computer vision task of detecting instances (such as humans, buildings, or cars) in an image. Object detection models receive an image as input and output
coordinates of the bounding boxes and associated labels of the detected objects. An image can contain multiple objects,
each with its own bounding box and a label (e.g. it can have a car and a building), and each object can
be present in different parts of an image (e.g. the image can have several cars).
This task is commonly used in autonomous driving for detecting things like pedestrians, road signs, and traffic lights.
Other applications include counting objects in images, image search, and more.

In this guide, you will learn how to:

 1. Finetune [DETR](https://huggingface.co/docs/transformers/model_doc/detr), a model that combines a convolutional
 backbone with an encoder-decoder Transformer, on the [CPPE-5](https://huggingface.co/datasets/cppe-5)
 dataset.
 2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/object-detection)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q datasets transformers accelerate timm
pip install -q -U albumentations>=1.4.5 torchmetrics pycocotools
```

You'll use 🤗 Datasets to load a dataset from the Hugging Face Hub, 🤗 Transformers to train your model,
and `albumentations` to augment the data.

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

To get started, we'll define global constants, namely the model name and image size. For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.

```py
>>> MODEL_NAME = "microsoft/conditional-detr-resnet-50"  # or "facebook/detr-resnet-50"
>>> IMAGE_SIZE = 480
```

## Load the CPPE-5 dataset

The [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.


key: _node_content
string_value:
{"id_": "33619358-f15e-4b6b-9690-e5d07dd43871", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c70a1d54-d554-480c-a58e-0e80e535de01", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "24c348c4b9e935e1bed0ebfe970a96321405c12cca085b6feb6a4a4527b4cf92", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4abc7316-4d26-402c-8c9a-3948d7bc0216", "node_type": "1", "metadata": {}, "hash": "e42d4eb927780ad0d43cef68cecf1a16d035757554871ad9b7283b7aef6b443e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 993, "end_char_idx": 3318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/object-detection)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q datasets transformers accelerate timm
pip install -q -U albumentations>=1.4.5 torchmetrics pycocotools
```

You'll use 🤗 Datasets to load a dataset from the Hugging Face Hub, 🤗 Transformers to train your model,
and `albumentations` to augment the data.

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

To get started, we'll define global constants, namely the model name and image size. For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.

```py
>>> MODEL_NAME = "microsoft/conditional-detr-resnet-50"  # or "facebook/detr-resnet-50"
>>> IMAGE_SIZE = 480
```

## Load the CPPE-5 dataset

The [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.

Start by loading the dataset and creating a `validation` split from `train`:

```py
>>> from datasets import load_dataset

>>> cppe5 = load_dataset("cppe-5")

>>> if "validation" not in cppe5:
...     split = cppe5["train"].train_test_split(0.15, seed=1337)
...     cppe5["train"] = split["train"]
...     cppe5["validation"] = split["test"]

>>> cppe5
DatasetDict({
    train: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 850
    })
    test: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 29
    })
    validation: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 150
    })
})
```

You'll see that this dataset has 1000 images for train and validation sets and a test set with 29 images.

To get familiar with the data, explore what the examples look like.


key: _node_content
string_value:
{"id_": "4abc7316-4d26-402c-8c9a-3948d7bc0216", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "33619358-f15e-4b6b-9690-e5d07dd43871", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "380bd0ee80eeee7be2b81ba8e69aa0a7f92db4a659d500ba0ebf6ea84843e7d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77557ed8-0c9b-423e-ad72-8f54475c87cb", "node_type": "1", "metadata": {}, "hash": "d0a64fc7743877eee49195caa32541c90469e2b61e9763f75539e033815242cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1887, "end_char_idx": 4475, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For this tutorial, we'll use the conditional DETR model due to its faster convergence. Feel free to select any object detection model available in the `transformers` library.

```py
>>> MODEL_NAME = "microsoft/conditional-detr-resnet-50"  # or "facebook/detr-resnet-50"
>>> IMAGE_SIZE = 480
```

## Load the CPPE-5 dataset

The [CPPE-5 dataset](https://huggingface.co/datasets/cppe-5) contains images with
annotations identifying medical personal protective equipment (PPE) in the context of the COVID-19 pandemic.

Start by loading the dataset and creating a `validation` split from `train`:

```py
>>> from datasets import load_dataset

>>> cppe5 = load_dataset("cppe-5")

>>> if "validation" not in cppe5:
...     split = cppe5["train"].train_test_split(0.15, seed=1337)
...     cppe5["train"] = split["train"]
...     cppe5["validation"] = split["test"]

>>> cppe5
DatasetDict({
    train: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 850
    })
    test: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 29
    })
    validation: Dataset({
        features: ['image_id', 'image', 'width', 'height', 'objects'],
        num_rows: 150
    })
})
```

You'll see that this dataset has 1000 images for train and validation sets and a test set with 29 images.

To get familiar with the data, explore what the examples look like.

```py
>>> cppe5["train"][0]
{
  'image_id': 366,
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=500x290>,
  'width': 500,
  'height': 500,
  'objects': {
    'id': [1932, 1933, 1934],
    'area': [27063, 34200, 32431],
    'bbox': [[29.0, 11.0, 97.0, 279.0],
      [201.0, 1.0, 120.0, 285.0],
      [382.0, 0.0, 113.0, 287.0]],
    'category': [0, 0, 0]
  }
}
```

The examples in the dataset have the following fields:
- `image_id`: the example image id
- `image`: a `PIL.Image.Image` object containing the image
- `width`: width of the image
- `height`: height of the image
- `objects`: a dictionary containing bounding box metadata for the objects in the image:
  - `id`: the annotation id
  - `area`: the area of the bounding box
  - `bbox`: the object's bounding box (in the [COCO format](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#coco) )
  - `category`: the object's category, with possible values including `Coverall (0)`, `Face_Shield (1)`, `Gloves (2)`, `Goggles (3)` and `Mask (4)`

You may notice that the `bbox` field follows the COCO format, which is the format that the DETR model expects.


key: _node_content
string_value:
{"id_": "77557ed8-0c9b-423e-ad72-8f54475c87cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4abc7316-4d26-402c-8c9a-3948d7bc0216", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "0e194871d3220975a67f8d19e6009b876bec3a4ff01fb94b14d0eaf0080efeea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "597eeaa3-0e5c-4e63-9535-b3cd32e901d1", "node_type": "1", "metadata": {}, "hash": "76dcc5b53250fae83d0ef8ef2392f35fbdc19b1934b867946bb7c4b697025fe5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3320, "end_char_idx": 4760, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> cppe5["train"][0]
{
  'image_id': 366,
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=500x290>,
  'width': 500,
  'height': 500,
  'objects': {
    'id': [1932, 1933, 1934],
    'area': [27063, 34200, 32431],
    'bbox': [[29.0, 11.0, 97.0, 279.0],
      [201.0, 1.0, 120.0, 285.0],
      [382.0, 0.0, 113.0, 287.0]],
    'category': [0, 0, 0]
  }
}
```

The examples in the dataset have the following fields:
- `image_id`: the example image id
- `image`: a `PIL.Image.Image` object containing the image
- `width`: width of the image
- `height`: height of the image
- `objects`: a dictionary containing bounding box metadata for the objects in the image:
  - `id`: the annotation id
  - `area`: the area of the bounding box
  - `bbox`: the object's bounding box (in the [COCO format](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/#coco) )
  - `category`: the object's category, with possible values including `Coverall (0)`, `Face_Shield (1)`, `Gloves (2)`, `Goggles (3)` and `Mask (4)`

You may notice that the `bbox` field follows the COCO format, which is the format that the DETR model expects.
However, the grouping of the fields inside `objects` differs from the annotation format DETR requires. You will
need to apply some preprocessing transformations before using this data for training.

To get an even better understanding of the data, visualize an example in the dataset.


key: _node_content
string_value:
{"id_": "597eeaa3-0e5c-4e63-9535-b3cd32e901d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77557ed8-0c9b-423e-ad72-8f54475c87cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "afd437b4db21030193349a9b1d3123dfae00337a8edbb27f184aafefac17de74", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b6818bd-b948-4bbf-b94d-225d889e87e6", "node_type": "1", "metadata": {}, "hash": "7c29284067bf392a3b88dae6f2f99a9958154e4b108ac43616af81f2d9b5b4ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4476, "end_char_idx": 7433, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, the grouping of the fields inside `objects` differs from the annotation format DETR requires. You will
need to apply some preprocessing transformations before using this data for training.

To get an even better understanding of the data, visualize an example in the dataset.

```py
>>> import numpy as np
>>> import os
>>> from PIL import Image, ImageDraw

>>> image = cppe5["train"][2]["image"]
>>> annotations = cppe5["train"][2]["objects"]
>>> draw = ImageDraw.Draw(image)

>>> categories = cppe5["train"].features["objects"].feature["category"].names

>>> id2label = {index: x for index, x in enumerate(categories, start=0)}
>>> label2id = {v: k for k, v in id2label.items()}

>>> for i in range(len(annotations["id"])):
...     box = annotations["bbox"][i]
...     class_idx = annotations["category"][i]
...     x, y, w, h = tuple(box)
...     # Check if coordinates are normalized or not
...     if max(box) > 1.0:
...         # Coordinates are un-normalized, no need to re-scale them
...         x1, y1 = int(x), int(y)
...         x2, y2 = int(x + w), int(y + h)
...     else:
...         # Coordinates are normalized, re-scale them
...         x1 = int(x * width)
...         y1 = int(y * height)
...         x2 = int((x + w) * width)
...         y2 = int((y + h) * height)
...     draw.rectangle((x, y, x + w, y + h), outline="red", width=1)
...     draw.text((x, y), id2label[class_idx], fill="white")

>>> image
```
<div class="flex justify-center">
    <img src="https://i.imgur.com/oVQb9SF.png" alt="CPPE-5 Image Example"/>
</div>


To visualize the bounding boxes with associated labels, you can get the labels from the dataset's metadata, specifically
the `category` field.
You'll also want to create dictionaries that map a label id to a label class (`id2label`) and the other way around (`label2id`).
You can use them later when setting up the model. Including these maps will make your model reusable by others if you share
it on the Hugging Face Hub. Please note that, the part of above code that draws the bounding boxes assume that it is in `COCO` format `(x_min, y_min, width, height)`. It has to be adjusted to work for other formats like `(x_min, y_min, x_max, y_max)`.

As a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for
object detection is bounding boxes that "stretch" beyond the edge of the image. Such "runaway" bounding boxes can raise
errors during training and should be addressed. There are a few examples with this issue in this dataset.
To keep things simple in this guide, we will set `clip=True` for `BboxParams` in transformations below.

## Preprocess the data

To finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.
[`AutoImageProcessor`] takes care of processing image data to create `pixel_values`, `pixel_mask`, and
`labels` that a DETR model can train with.


key: _node_content
string_value:
{"id_": "2b6818bd-b948-4bbf-b94d-225d889e87e6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "597eeaa3-0e5c-4e63-9535-b3cd32e901d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "818de2b15eab10366b4d59a573084a81e747b9544aec8e635a20415d5f966710", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca5eb46d-5749-4ea8-a49b-3f8c289a295f", "node_type": "1", "metadata": {}, "hash": "c35e953b9512e753411c4004b1f2b62e16c042dfb529d7f77cc8dea243add8fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6033, "end_char_idx": 9114, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To visualize the bounding boxes with associated labels, you can get the labels from the dataset's metadata, specifically
the `category` field.
You'll also want to create dictionaries that map a label id to a label class (`id2label`) and the other way around (`label2id`).
You can use them later when setting up the model. Including these maps will make your model reusable by others if you share
it on the Hugging Face Hub. Please note that, the part of above code that draws the bounding boxes assume that it is in `COCO` format `(x_min, y_min, width, height)`. It has to be adjusted to work for other formats like `(x_min, y_min, x_max, y_max)`.

As a final step of getting familiar with the data, explore it for potential issues. One common problem with datasets for
object detection is bounding boxes that "stretch" beyond the edge of the image. Such "runaway" bounding boxes can raise
errors during training and should be addressed. There are a few examples with this issue in this dataset.
To keep things simple in this guide, we will set `clip=True` for `BboxParams` in transformations below.

## Preprocess the data

To finetune a model, you must preprocess the data you plan to use to match precisely the approach used for the pre-trained model.
[`AutoImageProcessor`] takes care of processing image data to create `pixel_values`, `pixel_mask`, and
`labels` that a DETR model can train with. The image processor has some attributes that you won't have to worry about:

- `image_mean = [0.485, 0.456, 0.406 ]`
- `image_std = [0.229, 0.224, 0.225]`

These are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial
to replicate when doing inference or finetuning a pre-trained image model.

Instantiate the image processor from the same checkpoint as the model you want to finetune.

```py
>>> from transformers import AutoImageProcessor

>>> MAX_SIZE = IMAGE_SIZE

>>> image_processor = AutoImageProcessor.from_pretrained(
...     MODEL_NAME,
...     do_resize=True,
...     size={"max_height": MAX_SIZE, "max_width": MAX_SIZE},
...     do_pad=True,
...     pad_size={"height": MAX_SIZE, "width": MAX_SIZE},
... )
```

Before passing the images to the `image_processor`, apply two preprocessing transformations to the dataset:
- Augmenting images
- Reformatting annotations to meet DETR expectations

First, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use [Albumentations](https://albumentations.ai/docs/).
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),
and it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).


key: _node_content
string_value:
{"id_": "ca5eb46d-5749-4ea8-a49b-3f8c289a295f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b6818bd-b948-4bbf-b94d-225d889e87e6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "730f0de02729a39c75f344e8828dcb59b434375c83bc2e854264a22d1be6c606", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7aad290e-d1c9-4219-9513-e42658ec3c9c", "node_type": "1", "metadata": {}, "hash": "e3d3408c28ef45c5a0f556e3d1aa06cacde680bf44f90dd18b57b2524787728a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7434, "end_char_idx": 10095, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The image processor has some attributes that you won't have to worry about:

- `image_mean = [0.485, 0.456, 0.406 ]`
- `image_std = [0.229, 0.224, 0.225]`

These are the mean and standard deviation used to normalize images during the model pre-training. These values are crucial
to replicate when doing inference or finetuning a pre-trained image model.

Instantiate the image processor from the same checkpoint as the model you want to finetune.

```py
>>> from transformers import AutoImageProcessor

>>> MAX_SIZE = IMAGE_SIZE

>>> image_processor = AutoImageProcessor.from_pretrained(
...     MODEL_NAME,
...     do_resize=True,
...     size={"max_height": MAX_SIZE, "max_width": MAX_SIZE},
...     do_pad=True,
...     pad_size={"height": MAX_SIZE, "width": MAX_SIZE},
... )
```

Before passing the images to the `image_processor`, apply two preprocessing transformations to the dataset:
- Augmenting images
- Reformatting annotations to meet DETR expectations

First, to make sure the model does not overfit on the training data, you can apply image augmentation with any data augmentation library. Here we use [Albumentations](https://albumentations.ai/docs/).
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),
and it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).

```py
>>> import albumentations as A

>>> train_augment_and_transform = A.Compose(
...     [
...         A.Perspective(p=0.1),
...         A.HorizontalFlip(p=0.5),
...         A.RandomBrightnessContrast(p=0.5),
...         A.HueSaturationValue(p=0.1),
...     ],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True, min_area=25),
... )

>>> validation_transform = A.Compose(
...     [A.NoOp()],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True),
... )
```

The `image_processor` expects the annotations to be in the following format: `{'image_id': int, 'annotations': List[Dict]}`,
 where each dictionary is a COCO object annotation. Let's add a function to reformat annotations for a single example:

```py
>>> def format_image_annotations_as_coco(image_id, categories, areas, bboxes):
...     """Format one set of image annotations to the COCO format

...     Args:
...         image_id (str): image id. e.g.


key: _node_content
string_value:
{"id_": "7aad290e-d1c9-4219-9513-e42658ec3c9c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca5eb46d-5749-4ea8-a49b-3f8c289a295f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "9ac2555fc8b6ad09d885f97628327ac3b4fdce20b7d4e5d14e2fe66cfa618cff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fac562e2-0a1a-4964-8f43-c3150ef9df3d", "node_type": "1", "metadata": {}, "hash": "b84914bda320b3f08e5e432afb8242660428b333d68de5ea81aa07cef3ea7ced", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8538, "end_char_idx": 12459, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here we use [Albumentations](https://albumentations.ai/docs/).
This library ensures that transformations affect the image and update the bounding boxes accordingly.
The 🤗 Datasets library documentation has a detailed [guide on how to augment images for object detection](https://huggingface.co/docs/datasets/object_detection),
and it uses the exact same dataset as an example. Apply some geometric and color transformations to the image. For additional augmentation options, explore the [Albumentations Demo Space](https://huggingface.co/spaces/qubvel-hf/albumentations-demo).

```py
>>> import albumentations as A

>>> train_augment_and_transform = A.Compose(
...     [
...         A.Perspective(p=0.1),
...         A.HorizontalFlip(p=0.5),
...         A.RandomBrightnessContrast(p=0.5),
...         A.HueSaturationValue(p=0.1),
...     ],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True, min_area=25),
... )

>>> validation_transform = A.Compose(
...     [A.NoOp()],
...     bbox_params=A.BboxParams(format="coco", label_fields=["category"], clip=True),
... )
```

The `image_processor` expects the annotations to be in the following format: `{'image_id': int, 'annotations': List[Dict]}`,
 where each dictionary is a COCO object annotation. Let's add a function to reformat annotations for a single example:

```py
>>> def format_image_annotations_as_coco(image_id, categories, areas, bboxes):
...     """Format one set of image annotations to the COCO format

...     Args:
...         image_id (str): image id. e.g. "0001"
...         categories (List[int]): list of categories/class labels corresponding to provided bounding boxes
...         areas (List[float]): list of corresponding areas to provided bounding boxes
...         bboxes (List[Tuple[float]]): list of bounding boxes provided in COCO format
...             ([center_x, center_y, width, height] in absolute coordinates)

...     Returns:
...         dict: {
...             "image_id": image id,
...             "annotations": list of formatted annotations
...         }
...     """
...     annotations = []
...     for category, area, bbox in zip(categories, areas, bboxes):
...         formatted_annotation = {
...             "image_id": image_id,
...             "category_id": category,
...             "iscrowd": 0,
...             "area": area,
...             "bbox": list(bbox),
...         }
...         annotations.append(formatted_annotation)

...     return {
...         "image_id": image_id,
...         "annotations": annotations,
...     }

```

Now you can combine the image and annotation transformations to use on a batch of examples:

```py
>>> def augment_and_transform_batch(examples, transform, image_processor, return_pixel_mask=False):
...     """Apply augmentations and format annotations in COCO format for object detection task"""

...     images = []
...     annotations = []
...     for image_id, image, objects in zip(examples["image_id"], examples["image"], examples["objects"]):
...         image = np.array(image.convert("RGB"))

...         # apply augmentations
...         output = transform(image=image, bboxes=objects["bbox"], category=objects["category"])
...         images.append(output["image"])

...         # format annotations in COCO format
...         formatted_annotations = format_image_annotations_as_coco(
...             image_id, output["category"], objects["area"], output["bboxes"]
...         )
...         annotations.append(formatted_annotations)

...     # Apply the image processor transformations: resizing, rescaling, normalization
...     result = image_processor(images=images, annotations=annotations, return_tensors="pt")

...     if not return_pixel_mask:
...         result.pop("pixel_mask", None)

...     return result
```

Apply this preprocessing function to the entire dataset using 🤗 Datasets [`~datasets.Dataset.with_transform`] method.


key: _node_content
string_value:
{"id_": "fac562e2-0a1a-4964-8f43-c3150ef9df3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7aad290e-d1c9-4219-9513-e42658ec3c9c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "59180481fa7f66ddb8ab07845c7115d29e89ae33ff074b64d539bdaece7bc47f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94d5a71e-eb5c-4690-8ac9-306b117c1772", "node_type": "1", "metadata": {}, "hash": "16362efe4a5f5d6d55d670328bfffcfbc9133716ef95184389c0dc299ad84d30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This method applies
transformations on the fly when you load an element of the dataset.

At this point, you can check what an example from the dataset looks like after the transformations. You should see a tensor
with `pixel_values`, a tensor with `pixel_mask`, and `labels`.

```py
>>> from functools import partial

>>> # Make transform functions for batch and apply for dataset splits
>>> train_transform_batch = partial(
.     augment_and_transform_batch, transform=train_augment_and_transform, image_processor=image_processor
. )
>>> validation_transform_batch = partial(
.     augment_and_transform_batch, transform=validation_transform, image_processor=image_processor
. )

>>> cppe5["train"] = cppe5["train"].with_transform(train_transform_batch)
>>> cppe5["validation"] = cppe5["validation"].with_transform(validation_transform_batch)
>>> cppe5["test"] = cppe5["test"].with_transform(validation_transform_batch)

>>> cppe5["train"][15]
{'pixel_values': tensor([[[ 1.9235,  1.9407,  1.9749,  . -0.7822, -0.7479, -0.6965],
          [ 1.9578,  1.9749,  1.9920,  . -0.7993, -0.7650, -0.7308],
          [ 2.0092,  2.0092,  2.0263,  . -0.8507, -0.8164, -0.7822],
          .
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741]],
  
          [[ 1.6232,  1.6408,  1.6583,  .  0.8704,  1.0105,  1.1331],
          [ 1.6408,  1.6583,  1.6758,  .  0.8529,  0.9930,  1.0980],
          [ 1.6933,  1.6933,  1.7108,  .  0.8179,  0.9580,  1.0630],
          .
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052]],
  
          [[ 1.8905,  1.9080,  1.9428,  .


key: _node_content
string_value:
{"id_": "94d5a71e-eb5c-4690-8ac9-306b117c1772", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fac562e2-0a1a-4964-8f43-c3150ef9df3d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "7c929d3d0046e744b512de650353d4f40865dbdbc682f012b26f6c7494ade711", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d803773-6115-4400-8784-408326e5622b", "node_type": "1", "metadata": {}, "hash": "18d08ee815e7860e2de3464b851dacbeb4a07d76cc200bc0f4f0ffa7f265a47e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741],
          [ 0.0741,  0.0741,  0.0741,  .  0.0741,  0.0741,  0.0741]],
  
          [[ 1.6232,  1.6408,  1.6583,  .  0.8704,  1.0105,  1.1331],
          [ 1.6408,  1.6583,  1.6758,  .  0.8529,  0.9930,  1.0980],
          [ 1.6933,  1.6933,  1.7108,  .  0.8179,  0.9580,  1.0630],
          .
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052],
          [ 0.2052,  0.2052,  0.2052,  .  0.2052,  0.2052,  0.2052]],
  
          [[ 1.8905,  1.9080,  1.9428,  . -0.1487, -0.0964, -0.0615],
          [ 1.9254,  1.9428,  1.9603,  . -0.1661, -0.1138, -0.0790],
          [ 1.9777,  1.9777,  1.9951,  . -0.2010, -0.1138, -0.0790],
          .
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265]]]),
  'labels': {'image_id': tensor([688]), 'class_labels': tensor([3, 4, 2, 0, 0]), 'boxes': tensor([[0.4700, 0.1933, 0.1467, 0.0767],
          [0.4858, 0.2600, 0.1150, 0.1000],
          [0.4042, 0.4517, 0.1217, 0.1300],
          [0.4242, 0.3217, 0.3617, 0.5567],


key: _node_content
string_value:
{"id_": "9d803773-6115-4400-8784-408326e5622b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94d5a71e-eb5c-4690-8ac9-306b117c1772", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "68dad91b4728176f2c80cfe129df39858878409261e7a8cb8bea766653e20187", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d73358d6-82ef-4596-8acc-001b96cb0431", "node_type": "1", "metadata": {}, "hash": "e58d179bc248edbe2e04351c151c238e22920aa8707ea729758619aa449148a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9080,  1.9428,  . -0.1487, -0.0964, -0.0615],
          [ 1.9254,  1.9428,  1.9603,  . -0.1661, -0.1138, -0.0790],
          [ 1.9777,  1.9777,  1.9951,  . -0.2010, -0.1138, -0.0790],
          .
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265],
          [ 0.4265,  0.4265,  0.4265,  .  0.4265,  0.4265,  0.4265]]]),
  'labels': {'image_id': tensor([688]), 'class_labels': tensor([3, 4, 2, 0, 0]), 'boxes': tensor([[0.4700, 0.1933, 0.1467, 0.0767],
          [0.4858, 0.2600, 0.1150, 0.1000],
          [0.4042, 0.4517, 0.1217, 0.1300],
          [0.4242, 0.3217, 0.3617, 0.5567],
          [0.6617, 0.4033, 0.5400, 0.4533]]), 'area': tensor([ 4048.  4140.  5694. 72478. 88128.]), 'iscrowd': tensor([0, 0, 0, 0, 0]), 'orig_size': tensor([480, 480])}}
```

You have successfully augmented the individual images and prepared their annotations. However, preprocessing isn't
complete yet. In the final step, create a custom `collate_fn` to batch images together.
Pad images (which are now `pixel_values`) to the largest image in a batch, and create a corresponding `pixel_mask`
to indicate which pixels are real (1) and which are padding (0).

```py
>>> import torch

>>> def collate_fn(batch):
...     data = {}
...     data["pixel_values"] = torch.stack([x["pixel_values"] for x in batch])
...     data["labels"] = [x["labels"] for x in batch]
...     if "pixel_mask" in batch[0]:
...         data["pixel_mask"] = torch.stack([x["pixel_mask"] for x in batch])
...     return data

```

## Preparing function to compute mAP

Object detection models are commonly evaluated with a set of <a href="https://cocodataset.org/#detection-eval">COCO-style metrics</a>. We are going to use `torchmetrics` to compute `mAP` (mean average precision) and `mAR` (mean average recall) metrics and will wrap it to `compute_metrics` function in order to use in [`Trainer`] for evaluation.


key: _node_content
string_value:
{"id_": "d73358d6-82ef-4596-8acc-001b96cb0431", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d803773-6115-4400-8784-408326e5622b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f13890e4f4f4b24d274cc35e95d441e362aa1a359439b8d0fa6c8a38fff9ead5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2d7a32a-557b-4e24-881a-1e5571d041bd", "node_type": "1", "metadata": {}, "hash": "e8654c2503d832dfc38d850656525f4459f74a861f477d5a338907750ae2e01f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.1300],
          [0.4242, 0.3217, 0.3617, 0.5567],
          [0.6617, 0.4033, 0.5400, 0.4533]]), 'area': tensor([ 4048.  4140.  5694. 72478. 88128.]), 'iscrowd': tensor([0, 0, 0, 0, 0]), 'orig_size': tensor([480, 480])}}
```

You have successfully augmented the individual images and prepared their annotations. However, preprocessing isn't
complete yet. In the final step, create a custom `collate_fn` to batch images together.
Pad images (which are now `pixel_values`) to the largest image in a batch, and create a corresponding `pixel_mask`
to indicate which pixels are real (1) and which are padding (0).

```py
>>> import torch

>>> def collate_fn(batch):
...     data = {}
...     data["pixel_values"] = torch.stack([x["pixel_values"] for x in batch])
...     data["labels"] = [x["labels"] for x in batch]
...     if "pixel_mask" in batch[0]:
...         data["pixel_mask"] = torch.stack([x["pixel_mask"] for x in batch])
...     return data

```

## Preparing function to compute mAP

Object detection models are commonly evaluated with a set of <a href="https://cocodataset.org/#detection-eval">COCO-style metrics</a>. We are going to use `torchmetrics` to compute `mAP` (mean average precision) and `mAR` (mean average recall) metrics and will wrap it to `compute_metrics` function in order to use in [`Trainer`] for evaluation.

Intermediate format of boxes used for training is `YOLO` (normalized) but we will compute metrics for boxes in `Pascal VOC` (absolute) format in order to correctly handle box areas. Let's define a function that converts bounding boxes to `Pascal VOC` format:

```py
>>> from transformers.image_transforms import center_to_corners_format

>>> def convert_bbox_yolo_to_pascal(boxes, image_size):
...     """
...     Convert bounding boxes from YOLO format (x_center, y_center, width, height) in range [0, 1]
...     to Pascal VOC format (x_min, y_min, x_max, y_max) in absolute coordinates.

...     Args:
...         boxes (torch.Tensor): Bounding boxes in YOLO format
...         image_size (Tuple[int, int]): Image size in format (height, width)

...     Returns:
...         torch.Tensor: Bounding boxes in Pascal VOC format (x_min, y_min, x_max, y_max)
...     """
...     # convert center to corners format
...     boxes = center_to_corners_format(boxes)

...     # convert to absolute coordinates
...     height, width = image_size
...     boxes = boxes * torch.tensor([[width, height, width, height]])

...     return boxes
```

Then, in `compute_metrics` function we collect `predicted` and `target` bounding boxes, scores and labels from evaluation loop results and pass it to the scoring function.

```py
>>> import numpy as np
>>> from dataclasses import dataclass
>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision


>>> @dataclass
>>> class ModelOutput:
...     logits: torch.Tensor
...     pred_boxes: torch.Tensor


key: _node_content
string_value:
{"id_": "d2d7a32a-557b-4e24-881a-1e5571d041bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d73358d6-82ef-4596-8acc-001b96cb0431", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "070ed92bb3a30c34e81d3e65d52f9920cf098942413d17cb165e25e0d9bcc94d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "66a5e867-942f-4f91-84ce-037f11ccc8be", "node_type": "1", "metadata": {}, "hash": "1ffb0299a3d99bff0a66034bb4407641898738e0867c0aa7315f300ca0ea0d4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16339, "end_char_idx": 21366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Intermediate format of boxes used for training is `YOLO` (normalized) but we will compute metrics for boxes in `Pascal VOC` (absolute) format in order to correctly handle box areas. Let's define a function that converts bounding boxes to `Pascal VOC` format:

```py
>>> from transformers.image_transforms import center_to_corners_format

>>> def convert_bbox_yolo_to_pascal(boxes, image_size):
...     """
...     Convert bounding boxes from YOLO format (x_center, y_center, width, height) in range [0, 1]
...     to Pascal VOC format (x_min, y_min, x_max, y_max) in absolute coordinates.

...     Args:
...         boxes (torch.Tensor): Bounding boxes in YOLO format
...         image_size (Tuple[int, int]): Image size in format (height, width)

...     Returns:
...         torch.Tensor: Bounding boxes in Pascal VOC format (x_min, y_min, x_max, y_max)
...     """
...     # convert center to corners format
...     boxes = center_to_corners_format(boxes)

...     # convert to absolute coordinates
...     height, width = image_size
...     boxes = boxes * torch.tensor([[width, height, width, height]])

...     return boxes
```

Then, in `compute_metrics` function we collect `predicted` and `target` bounding boxes, scores and labels from evaluation loop results and pass it to the scoring function.

```py
>>> import numpy as np
>>> from dataclasses import dataclass
>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision


>>> @dataclass
>>> class ModelOutput:
...     logits: torch.Tensor
...     pred_boxes: torch.Tensor


>>> @torch.no_grad()
>>> def compute_metrics(evaluation_results, image_processor, threshold=0.0, id2label=None):
...     """
...     Compute mean average mAP, mAR and their variants for the object detection task.

...     Args:
...         evaluation_results (EvalPrediction): Predictions and targets from evaluation.
...         threshold (float, optional): Threshold to filter predicted boxes by confidence. Defaults to 0.0.
...         id2label (Optional[dict], optional): Mapping from class id to class name. Defaults to None.

...     Returns:
...         Mapping[str, float]: Metrics in a form of dictionary {<metric_name>: <metric_value>}
...     """

...     predictions, targets = evaluation_results.predictions, evaluation_results.label_ids

...     # For metric computation we need to provide:
...     #  - targets in a form of list of dictionaries with keys "boxes", "labels"
...     #  - predictions in a form of list of dictionaries with keys "boxes", "scores", "labels"

...     image_sizes = []
...     post_processed_targets = []
...     post_processed_predictions = []

...     # Collect targets in the required format for metric computation
...     for batch in targets:
...         # collect image sizes, we will need them for predictions post processing
...         batch_image_sizes = torch.tensor(np.array([x["orig_size"] for x in batch]))
...         image_sizes.append(batch_image_sizes)
...         # collect targets in the required format for metric computation
...         # boxes were converted to YOLO format needed for model training
...         # here we will convert them to Pascal VOC format (x_min, y_min, x_max, y_max)
...         for image_target in batch:
...             boxes = torch.tensor(image_target["boxes"])
...             boxes = convert_bbox_yolo_to_pascal(boxes, image_target["orig_size"])
...             labels = torch.tensor(image_target["class_labels"])
...             post_processed_targets.append({"boxes": boxes, "labels": labels})

...     # Collect predictions in the required format for metric computation,
...     # model produce boxes in YOLO format, then image_processor convert them to Pascal VOC format
...     for batch, target_sizes in zip(predictions, image_sizes):
...         batch_logits, batch_boxes = batch[1], batch[2]
...         output = ModelOutput(logits=torch.tensor(batch_logits), pred_boxes=torch.tensor(batch_boxes))
...         post_processed_output = image_processor.post_process_object_detection(
...             output, threshold=threshold, target_sizes=target_sizes
...         )
...         post_processed_predictions.extend(post_processed_output)

...     # Compute metrics
...     metric = MeanAveragePrecision(box_format="xyxy", class_metrics=True)
...     metric.update(post_processed_predictions, post_processed_targets)
...     metrics = metric.compute()

...     # Replace list of per class metrics with separate metric for each class
...     classes = metrics.pop("classes")
...     map_per_class = metrics.pop("map_per_class")
...     mar_100_per_class = metrics.pop("mar_100_per_class")
...     for class_id, class_map, class_mar in zip(classes, map_per_class, mar_100_per_class):
...         class_name = id2label[class_id.item()] if id2label is not None else class_id.item()
...         metrics[f"map_{class_name}"] = class_map
...         metrics[f"mar_100_{class_name}"] = class_mar

...     metrics = {k: round(v.item(), 4) for k, v in metrics.items()}

...     return metrics


key: _node_content
string_value:
{"id_": "66a5e867-942f-4f91-84ce-037f11ccc8be", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2d7a32a-557b-4e24-881a-1e5571d041bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "98094e923fc8e7f874a1ab59bac002494f4313ed427aea70c0a16cc304eadd4b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7d6d377-2c2e-4cc2-82bd-a9996c6b67db", "node_type": "1", "metadata": {}, "hash": "203b9167d068489e551c4d0a452a1bf3986c1b59b1342a2e5024b962b79e769e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> eval_compute_metrics_fn = partial(
...     compute_metrics, image_processor=image_processor, id2label=id2label, threshold=0.0
... )
```

## Training the detection model

You have done most of the heavy lifting in the previous sections, so now you are ready to train your model!
The images in this dataset are still quite large, even after resizing. This means that finetuning this model will
require at least one GPU.

Training involves the following steps:
1. Load the model with [`AutoModelForObjectDetection`] using the same checkpoint as in the preprocessing.
2. Define your training hyperparameters in [`TrainingArguments`].
3. Pass the training arguments to [`Trainer`] along with the model, dataset, image processor, and data collator.
4. Call [`~Trainer.train`] to finetune your model.

When loading the model from the same checkpoint that you used for the preprocessing, remember to pass the `label2id`
and `id2label` maps that you created earlier from the dataset's metadata. Additionally, we specify `ignore_mismatched_sizes=True` to replace the existing classification head with a new one.

```py
>>> from transformers import AutoModelForObjectDetection

>>> model = AutoModelForObjectDetection.from_pretrained(
...     MODEL_NAME,
...     id2label=id2label,
...     label2id=label2id,
...     ignore_mismatched_sizes=True,
... )
```

In the [`TrainingArguments`] use `output_dir` to specify where to save your model, then configure hyperparameters as you see fit. For `num_train_epochs=30` training will take about 35 minutes in Google Colab T4 GPU, increase the number of epoch to get better results.

Important notes:
 - Do not remove unused columns because this will drop the image column. Without the image column, you
can't create `pixel_values`. For this reason, set `remove_unused_columns` to `False`.
 - Set `eval_do_concat_batches=False` to get proper evaluation results. Images have different number of target boxes, if batches are concatenated we will not be able to determine which boxes belongs to particular image.

If you wish to share your model by pushing to the Hub, set `push_to_hub` to `True` (you must be signed in to Hugging
Face to upload your model).

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(
.     output_dir="detr_finetuned_cppe5",
.     num_train_epochs=30,
.     fp16=False,
.     per_device_train_batch_size=8,
.     dataloader_num_workers=4,
.     learning_rate=5e-5,
.     lr_scheduler_type="cosine",
.     weight_decay=1e-4,
.     max_grad_norm=0.01,
.     metric_for_best_model="eval_map",
.     greater_is_better=True,
.     load_best_model_at_end=True,
.     eval_strategy="epoch",
.     save_strategy="epoch",
.     save_total_limit=2,
.     remove_unused_columns=False,
.     eval_do_concat_batches=False,
.     push_to_hub=True,
. )
```

Finally, bring everything together, and call [`~transformers.Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
.     model=model,
.     args=training_args,
.     train_dataset=cppe5["train"],
.     eval_dataset=cppe5["validation"],
.     tokenizer=image_processor,
.     data_collator=collate_fn,
.


key: _node_content
string_value:
{"id_": "d7d6d377-2c2e-4cc2-82bd-a9996c6b67db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66a5e867-942f-4f91-84ce-037f11ccc8be", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "6bf2b8d18c63410b01789ea7033b8d17fcd600532520c2ee13b2a2ca4601362e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac4cfddd-0bc5-4648-84d2-ebbaef4488ae", "node_type": "1", "metadata": {}, "hash": "373bc6419be3b6ecc65f9036aaa36cd93684e1f2bfa34c2b0a662440b84d8ffc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Without the image column, you
can't create `pixel_values`. For this reason, set `remove_unused_columns` to `False`.
 - Set `eval_do_concat_batches=False` to get proper evaluation results. Images have different number of target boxes, if batches are concatenated we will not be able to determine which boxes belongs to particular image.

If you wish to share your model by pushing to the Hub, set `push_to_hub` to `True` (you must be signed in to Hugging
Face to upload your model).

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(
.     output_dir="detr_finetuned_cppe5",
.     num_train_epochs=30,
.     fp16=False,
.     per_device_train_batch_size=8,
.     dataloader_num_workers=4,
.     learning_rate=5e-5,
.     lr_scheduler_type="cosine",
.     weight_decay=1e-4,
.     max_grad_norm=0.01,
.     metric_for_best_model="eval_map",
.     greater_is_better=True,
.     load_best_model_at_end=True,
.     eval_strategy="epoch",
.     save_strategy="epoch",
.     save_total_limit=2,
.     remove_unused_columns=False,
.     eval_do_concat_batches=False,
.     push_to_hub=True,
. )
```

Finally, bring everything together, and call [`~transformers.Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
.     model=model,
.     args=training_args,
.     train_dataset=cppe5["train"],
.     eval_dataset=cppe5["validation"],
.     tokenizer=image_processor,
.     data_collator=collate_fn,
.     compute_metrics=eval_compute_metrics_fn,
. )

>>> trainer.train()
```
<div>

  <progress value='3210' max='3210' style='width:300px; height:20px; vertical-align: middle;'></progress>
  [3210/3210 26:07, Epoch 30/30]
</div>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>Epoch</th>
      <th>Training Loss</th>
      <th>Validation Loss</th>
      <th>Map</th>
      <th>Map 50</th>
      <th>Map 75</th>
      <th>Map Small</th>
      <th>Map Medium</th>
      <th>Map Large</th>
      <th>Mar 1</th>
      <th>Mar 10</th>
      <th>Mar 100</th>
      <th>Mar Small</th>
      <th>Mar Medium</th>
      <th>Mar Large</th>
      <th>Map Coverall</th>
      <th>Mar 100 Coverall</th>
      <th>Map Face Shield</th>
      <th>Mar 100 Face Shield</th>
      <th>Map Gloves</th>
      <th>Mar 100 Gloves</th>
      <th>Map Goggles</th>
      <th>Mar 100 Goggles</th>
      <th>Map Mask</th>
      <th>Mar 100 Mask</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td>No log</td>
      <td>2.


key: _node_content
string_value:
{"id_": "ac4cfddd-0bc5-4648-84d2-ebbaef4488ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7d6d377-2c2e-4cc2-82bd-a9996c6b67db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "111a8ec6a13c226afa5adf5bf0ced381e4f8c8d45f1deb0c6fd070e5c694da84", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4571cf4f-bd82-483d-b7dc-5ef8b2167454", "node_type": "1", "metadata": {}, "hash": "dc196baa4caa78fcd9ab800ea903b434a17e5fd60220550f72bbfe2837bff800", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
tokenizer=image_processor,
.     data_collator=collate_fn,
.     compute_metrics=eval_compute_metrics_fn,
. )

>>> trainer.train()
```
<div>

  <progress value='3210' max='3210' style='width:300px; height:20px; vertical-align: middle;'></progress>
  [3210/3210 26:07, Epoch 30/30]
</div>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>Epoch</th>
      <th>Training Loss</th>
      <th>Validation Loss</th>
      <th>Map</th>
      <th>Map 50</th>
      <th>Map 75</th>
      <th>Map Small</th>
      <th>Map Medium</th>
      <th>Map Large</th>
      <th>Mar 1</th>
      <th>Mar 10</th>
      <th>Mar 100</th>
      <th>Mar Small</th>
      <th>Mar Medium</th>
      <th>Mar Large</th>
      <th>Map Coverall</th>
      <th>Mar 100 Coverall</th>
      <th>Map Face Shield</th>
      <th>Mar 100 Face Shield</th>
      <th>Map Gloves</th>
      <th>Mar 100 Gloves</th>
      <th>Map Goggles</th>
      <th>Mar 100 Goggles</th>
      <th>Map Mask</th>
      <th>Mar 100 Mask</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td>No log</td>
      <td>2.629903</td>
      <td>0.008900</td>
      <td>0.023200</td>
      <td>0.006500</td>
      <td>0.001300</td>
      <td>0.002800</td>
      <td>0.020500</td>
      <td>0.021500</td>
      <td>0.070400</td>
      <td>0.101400</td>
      <td>0.007600</td>
      <td>0.106200</td>
      <td>0.096100</td>
      <td>0.036700</td>
      <td>0.232000</td>
      <td>0.000300</td>
      <td>0.019000</td>
      <td>0.003900</td>
      <td>0.125400</td>
      <td>0.000100</td>
      <td>0.003100</td>
      <td>0.003500</td>
      <td>0.127600</td>
    </tr>
    <tr>
      <td>2</td>
      <td>No log</td>
      <td>3.479864</td>
      <td>0.014800</td>
      <td>0.034600</td>
      <td>0.010800</td>
      <td>0.008600</td>
      <td>0.011700</td>
      <td>0.012500</td>
      <td>0.041100</td>
      <td>0.098700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "4571cf4f-bd82-483d-b7dc-5ef8b2167454", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac4cfddd-0bc5-4648-84d2-ebbaef4488ae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "9d50a3aaf633a4e555c6fe353601b9a892e4eb42b37767427958ead86182200d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ce97cee-686a-455f-bd05-85ed7ef903d1", "node_type": "1", "metadata": {}, "hash": "1afda9e8084f56c73b59c29a680d3ccd38ab76d71fac65497168968824f97bb7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25657, "end_char_idx": 27357, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
629903</td>
      <td>0.008900</td>
      <td>0.023200</td>
      <td>0.006500</td>
      <td>0.001300</td>
      <td>0.002800</td>
      <td>0.020500</td>
      <td>0.021500</td>
      <td>0.070400</td>
      <td>0.101400</td>
      <td>0.007600</td>
      <td>0.106200</td>
      <td>0.096100</td>
      <td>0.036700</td>
      <td>0.232000</td>
      <td>0.000300</td>
      <td>0.019000</td>
      <td>0.003900</td>
      <td>0.125400</td>
      <td>0.000100</td>
      <td>0.003100</td>
      <td>0.003500</td>
      <td>0.127600</td>
    </tr>
    <tr>
      <td>2</td>
      <td>No log</td>
      <td>3.479864</td>
      <td>0.014800</td>
      <td>0.034600</td>
      <td>0.010800</td>
      <td>0.008600</td>
      <td>0.011700</td>
      <td>0.012500</td>
      <td>0.041100</td>
      <td>0.098700</td>
      <td>0.130000</td>
      <td>0.056000</td>
      <td>0.062200</td>
      <td>0.111900</td>
      <td>0.053500</td>
      <td>0.447300</td>
      <td>0.010600</td>
      <td>0.100000</td>
      <td>0.000200</td>
      <td>0.022800</td>
      <td>0.000100</td>
      <td>0.015400</td>
      <td>0.009700</td>
      <td>0.064400</td>
    </tr>
    <tr>
      <td>3</td>
      <td>No log</td>
      <td>2.107622</td>
      <td>0.041700</td>
      <td>0.094000</td>
      <td>0.034300</td>
      <td>0.024100</td>
      <td>0.026400</td>
      <td>0.047400</td>
      <td>0.091500</td>
      <td>0.182800</td>
      <td>0.225800</td>
      <td>0.087200</td>
      <td>0.199400</td>
      <td>0.210600</td>
      <td>0.150900</td>
      <td>0.571200</td>
      <td>0.017300</td>
      <td>0.101300</td>
      <td>0.007300</td>
      <td>0.180400</td>
      <td>0.002100</td>
      <td>0.


key: _node_content
string_value:
{"id_": "2ce97cee-686a-455f-bd05-85ed7ef903d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4571cf4f-bd82-483d-b7dc-5ef8b2167454", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "5fbcefe4ad8170ddb9a307deb7391f3590f9f13318637796adf201c7912de1df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "656e5561-7b72-412e-90b7-9648122891ae", "node_type": "1", "metadata": {}, "hash": "8b7cfa94044ef4f91b74b20d79240bf61a7924ddf481aa1e238bb3796065f639", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26483, "end_char_idx": 28195, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
130000</td>
      <td>0.056000</td>
      <td>0.062200</td>
      <td>0.111900</td>
      <td>0.053500</td>
      <td>0.447300</td>
      <td>0.010600</td>
      <td>0.100000</td>
      <td>0.000200</td>
      <td>0.022800</td>
      <td>0.000100</td>
      <td>0.015400</td>
      <td>0.009700</td>
      <td>0.064400</td>
    </tr>
    <tr>
      <td>3</td>
      <td>No log</td>
      <td>2.107622</td>
      <td>0.041700</td>
      <td>0.094000</td>
      <td>0.034300</td>
      <td>0.024100</td>
      <td>0.026400</td>
      <td>0.047400</td>
      <td>0.091500</td>
      <td>0.182800</td>
      <td>0.225800</td>
      <td>0.087200</td>
      <td>0.199400</td>
      <td>0.210600</td>
      <td>0.150900</td>
      <td>0.571200</td>
      <td>0.017300</td>
      <td>0.101300</td>
      <td>0.007300</td>
      <td>0.180400</td>
      <td>0.002100</td>
      <td>0.026200</td>
      <td>0.031000</td>
      <td>0.250200</td>
    </tr>
    <tr>
      <td>4</td>
      <td>No log</td>
      <td>2.031242</td>
      <td>0.055900</td>
      <td>0.120600</td>
      <td>0.046900</td>
      <td>0.013800</td>
      <td>0.038100</td>
      <td>0.090300</td>
      <td>0.105900</td>
      <td>0.225600</td>
      <td>0.266100</td>
      <td>0.130200</td>
      <td>0.228100</td>
      <td>0.330000</td>
      <td>0.191000</td>
      <td>0.572100</td>
      <td>0.010600</td>
      <td>0.157000</td>
      <td>0.014600</td>
      <td>0.235300</td>
      <td>0.001700</td>
      <td>0.052300</td>
      <td>0.061800</td>
      <td>0.313800</td>
    </tr>
    <tr>
      <td>5</td>
      <td>3.889400</td>
      <td>1.883433</td>
      <td>0.089700</td>
      <td>0.201800</td>
      <td>0.067300</td>
      <td>0.


key: _node_content
string_value:
{"id_": "656e5561-7b72-412e-90b7-9648122891ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ce97cee-686a-455f-bd05-85ed7ef903d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "12b50bdfda5e8efdbf6a3d403fe4817cbe4e0503557ebb79529f596c10e7ccf9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "81677573-e93f-48ed-9d01-af34718df1c9", "node_type": "1", "metadata": {}, "hash": "3b4c3c66824a14a9f17aa3b355c14615390b67ba46b5293a1cf0b1e7ede648c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27333, "end_char_idx": 29047, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
002100</td>
      <td>0.026200</td>
      <td>0.031000</td>
      <td>0.250200</td>
    </tr>
    <tr>
      <td>4</td>
      <td>No log</td>
      <td>2.031242</td>
      <td>0.055900</td>
      <td>0.120600</td>
      <td>0.046900</td>
      <td>0.013800</td>
      <td>0.038100</td>
      <td>0.090300</td>
      <td>0.105900</td>
      <td>0.225600</td>
      <td>0.266100</td>
      <td>0.130200</td>
      <td>0.228100</td>
      <td>0.330000</td>
      <td>0.191000</td>
      <td>0.572100</td>
      <td>0.010600</td>
      <td>0.157000</td>
      <td>0.014600</td>
      <td>0.235300</td>
      <td>0.001700</td>
      <td>0.052300</td>
      <td>0.061800</td>
      <td>0.313800</td>
    </tr>
    <tr>
      <td>5</td>
      <td>3.889400</td>
      <td>1.883433</td>
      <td>0.089700</td>
      <td>0.201800</td>
      <td>0.067300</td>
      <td>0.022800</td>
      <td>0.065300</td>
      <td>0.129500</td>
      <td>0.136000</td>
      <td>0.272200</td>
      <td>0.303700</td>
      <td>0.112900</td>
      <td>0.312500</td>
      <td>0.424600</td>
      <td>0.300200</td>
      <td>0.585100</td>
      <td>0.032700</td>
      <td>0.202500</td>
      <td>0.031300</td>
      <td>0.271000</td>
      <td>0.008700</td>
      <td>0.126200</td>
      <td>0.075500</td>
      <td>0.333800</td>
    </tr>
    <tr>
      <td>6</td>
      <td>3.889400</td>
      <td>1.807503</td>
      <td>0.118500</td>
      <td>0.270900</td>
      <td>0.090200</td>
      <td>0.034900</td>
      <td>0.076700</td>
      <td>0.152500</td>
      <td>0.146100</td>
      <td>0.297800</td>
      <td>0.325400</td>
      <td>0.171700</td>
      <td>0.283700</td>
      <td>0.545900</td>
      <td>0.396900</td>
      <td>0.


key: _node_content
string_value:
{"id_": "81677573-e93f-48ed-9d01-af34718df1c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "656e5561-7b72-412e-90b7-9648122891ae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "6b0a06e8774a8eb5ad07a3af7e3aa536465c802ef02f208688957ca4c4fd877f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bcf1f395-b6c5-46e9-9ebd-4550de30cb08", "node_type": "1", "metadata": {}, "hash": "75a3afecbc9b6e80a73db81df48cef2bdaa36ae66b4e416302f5d8e49db21ee0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28195, "end_char_idx": 29851, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
022800</td>
      <td>0.065300</td>
      <td>0.129500</td>
      <td>0.136000</td>
      <td>0.272200</td>
      <td>0.303700</td>
      <td>0.112900</td>
      <td>0.312500</td>
      <td>0.424600</td>
      <td>0.300200</td>
      <td>0.585100</td>
      <td>0.032700</td>
      <td>0.202500</td>
      <td>0.031300</td>
      <td>0.271000</td>
      <td>0.008700</td>
      <td>0.126200</td>
      <td>0.075500</td>
      <td>0.333800</td>
    </tr>
    <tr>
      <td>6</td>
      <td>3.889400</td>
      <td>1.807503</td>
      <td>0.118500</td>
      <td>0.270900</td>
      <td>0.090200</td>
      <td>0.034900</td>
      <td>0.076700</td>
      <td>0.152500</td>
      <td>0.146100</td>
      <td>0.297800</td>
      <td>0.325400</td>
      <td>0.171700</td>
      <td>0.283700</td>
      <td>0.545900</td>
      <td>0.396900</td>
      <td>0.554500</td>
      <td>0.043000</td>
      <td>0.262000</td>
      <td>0.054500</td>
      <td>0.271900</td>
      <td>0.020300</td>
      <td>0.230800</td>
      <td>0.077600</td>
      <td>0.308000</td>
    </tr>
    <tr>
      <td>7</td>
      <td>3.889400</td>
      <td>1.716169</td>
      <td>0.143500</td>
      <td>0.307700</td>
      <td>0.123200</td>
      <td>0.045800</td>
      <td>0.097800</td>
      <td>0.258300</td>
      <td>0.165300</td>
      <td>0.327700</td>
      <td>0.352600</td>
      <td>0.140900</td>
      <td>0.336700</td>
      <td>0.599400</td>
      <td>0.442900</td>
      <td>0.620700</td>
      <td>0.069400</td>
      <td>0.301300</td>
      <td>0.081600</td>
      <td>0.292000</td>
      <td>0.011000</td>
      <td>0.230800</td>
      <td>0.112700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "bcf1f395-b6c5-46e9-9ebd-4550de30cb08", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "81677573-e93f-48ed-9d01-af34718df1c9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "4568b9e1c1e83923c0f49cfd9704431e6c1bc7394f4f6b9a2f81af9b192f72c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c5a7d23-322f-490a-8ed3-7c126971bd20", "node_type": "1", "metadata": {}, "hash": "64019a65921d21cfaf5ee3b6a573f28a004379d59e306e86c1c550f37f3c6db6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28999, "end_char_idx": 30691, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
545900</td>
      <td>0.396900</td>
      <td>0.554500</td>
      <td>0.043000</td>
      <td>0.262000</td>
      <td>0.054500</td>
      <td>0.271900</td>
      <td>0.020300</td>
      <td>0.230800</td>
      <td>0.077600</td>
      <td>0.308000</td>
    </tr>
    <tr>
      <td>7</td>
      <td>3.889400</td>
      <td>1.716169</td>
      <td>0.143500</td>
      <td>0.307700</td>
      <td>0.123200</td>
      <td>0.045800</td>
      <td>0.097800</td>
      <td>0.258300</td>
      <td>0.165300</td>
      <td>0.327700</td>
      <td>0.352600</td>
      <td>0.140900</td>
      <td>0.336700</td>
      <td>0.599400</td>
      <td>0.442900</td>
      <td>0.620700</td>
      <td>0.069400</td>
      <td>0.301300</td>
      <td>0.081600</td>
      <td>0.292000</td>
      <td>0.011000</td>
      <td>0.230800</td>
      <td>0.112700</td>
      <td>0.318200</td>
    </tr>
    <tr>
      <td>8</td>
      <td>3.889400</td>
      <td>1.679014</td>
      <td>0.153000</td>
      <td>0.355800</td>
      <td>0.127900</td>
      <td>0.038700</td>
      <td>0.115600</td>
      <td>0.291600</td>
      <td>0.176000</td>
      <td>0.322500</td>
      <td>0.349700</td>
      <td>0.135600</td>
      <td>0.326100</td>
      <td>0.643700</td>
      <td>0.431700</td>
      <td>0.582900</td>
      <td>0.069800</td>
      <td>0.265800</td>
      <td>0.088600</td>
      <td>0.274600</td>
      <td>0.028300</td>
      <td>0.280000</td>
      <td>0.146700</td>
      <td>0.345300</td>
    </tr>
    <tr>
      <td>9</td>
      <td>3.889400</td>
      <td>1.618239</td>
      <td>0.172100</td>
      <td>0.375300</td>
      <td>0.137600</td>
      <td>0.046100</td>
      <td>0.141700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "7c5a7d23-322f-490a-8ed3-7c126971bd20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bcf1f395-b6c5-46e9-9ebd-4550de30cb08", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "4cc6ba4273d12be3c818e3419744c2708a11e97d4fb644c68670a8e58e747b36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10b462d6-027f-4376-ae55-4116d99c6943", "node_type": "1", "metadata": {}, "hash": "f5ff7dbee939ba7d65841e11084bc55a2b84abb9e153701388ae71cbaba80e65", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29827, "end_char_idx": 31520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
112700</td>
      <td>0.318200</td>
    </tr>
    <tr>
      <td>8</td>
      <td>3.889400</td>
      <td>1.679014</td>
      <td>0.153000</td>
      <td>0.355800</td>
      <td>0.127900</td>
      <td>0.038700</td>
      <td>0.115600</td>
      <td>0.291600</td>
      <td>0.176000</td>
      <td>0.322500</td>
      <td>0.349700</td>
      <td>0.135600</td>
      <td>0.326100</td>
      <td>0.643700</td>
      <td>0.431700</td>
      <td>0.582900</td>
      <td>0.069800</td>
      <td>0.265800</td>
      <td>0.088600</td>
      <td>0.274600</td>
      <td>0.028300</td>
      <td>0.280000</td>
      <td>0.146700</td>
      <td>0.345300</td>
    </tr>
    <tr>
      <td>9</td>
      <td>3.889400</td>
      <td>1.618239</td>
      <td>0.172100</td>
      <td>0.375300</td>
      <td>0.137600</td>
      <td>0.046100</td>
      <td>0.141700</td>
      <td>0.308500</td>
      <td>0.194000</td>
      <td>0.356200</td>
      <td>0.386200</td>
      <td>0.162400</td>
      <td>0.359200</td>
      <td>0.677700</td>
      <td>0.469800</td>
      <td>0.623900</td>
      <td>0.102100</td>
      <td>0.317700</td>
      <td>0.099100</td>
      <td>0.290200</td>
      <td>0.029300</td>
      <td>0.335400</td>
      <td>0.160200</td>
      <td>0.364000</td>
    </tr>
    <tr>
      <td>10</td>
      <td>1.599700</td>
      <td>1.572512</td>
      <td>0.179500</td>
      <td>0.400400</td>
      <td>0.147200</td>
      <td>0.056500</td>
      <td>0.141700</td>
      <td>0.316700</td>
      <td>0.213100</td>
      <td>0.357600</td>
      <td>0.381300</td>
      <td>0.197900</td>
      <td>0.344300</td>
      <td>0.638500</td>
      <td>0.466900</td>
      <td>0.623900</td>
      <td>0.


key: _node_content
string_value:
{"id_": "10b462d6-027f-4376-ae55-4116d99c6943", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c5a7d23-322f-490a-8ed3-7c126971bd20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "adf721baf28ae411b2f75fed3f9fb49f28480c2f96089c02710ff4997a94d7d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9a62800a-722d-42bf-b3ee-b8fe518b9f8e", "node_type": "1", "metadata": {}, "hash": "ad49965ea3ab576c28faabe3ba243972bd7d18a408dcc325383b5db992d6bc03", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30667, "end_char_idx": 32362, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
141700</td>
      <td>0.308500</td>
      <td>0.194000</td>
      <td>0.356200</td>
      <td>0.386200</td>
      <td>0.162400</td>
      <td>0.359200</td>
      <td>0.677700</td>
      <td>0.469800</td>
      <td>0.623900</td>
      <td>0.102100</td>
      <td>0.317700</td>
      <td>0.099100</td>
      <td>0.290200</td>
      <td>0.029300</td>
      <td>0.335400</td>
      <td>0.160200</td>
      <td>0.364000</td>
    </tr>
    <tr>
      <td>10</td>
      <td>1.599700</td>
      <td>1.572512</td>
      <td>0.179500</td>
      <td>0.400400</td>
      <td>0.147200</td>
      <td>0.056500</td>
      <td>0.141700</td>
      <td>0.316700</td>
      <td>0.213100</td>
      <td>0.357600</td>
      <td>0.381300</td>
      <td>0.197900</td>
      <td>0.344300</td>
      <td>0.638500</td>
      <td>0.466900</td>
      <td>0.623900</td>
      <td>0.101300</td>
      <td>0.311400</td>
      <td>0.104700</td>
      <td>0.279500</td>
      <td>0.051600</td>
      <td>0.338500</td>
      <td>0.173000</td>
      <td>0.353300</td>
    </tr>
    <tr>
      <td>11</td>
      <td>1.599700</td>
      <td>1.528889</td>
      <td>0.192200</td>
      <td>0.415000</td>
      <td>0.160800</td>
      <td>0.053700</td>
      <td>0.150500</td>
      <td>0.378000</td>
      <td>0.211500</td>
      <td>0.371700</td>
      <td>0.397800</td>
      <td>0.204900</td>
      <td>0.374600</td>
      <td>0.684800</td>
      <td>0.491900</td>
      <td>0.632400</td>
      <td>0.131200</td>
      <td>0.346800</td>
      <td>0.122000</td>
      <td>0.300900</td>
      <td>0.038400</td>
      <td>0.344600</td>
      <td>0.177500</td>
      <td>0.364400</td>
    </tr>
    <tr>
      <td>12</td>
      <td>1.


key: _node_content
string_value:
{"id_": "9a62800a-722d-42bf-b3ee-b8fe518b9f8e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10b462d6-027f-4376-ae55-4116d99c6943", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "6f8b411c7e455f8ab9cf7e86bddd4268c5a983a639f4dbe3c845bfce443b61f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50d2eb20-db81-4a6d-b7f6-bdcd382ffd8e", "node_type": "1", "metadata": {}, "hash": "99bb9548732b8f93ee09d7a4ad21ed07efb9784166e641d8d8dc67900f39c06d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31496, "end_char_idx": 33191, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
623900</td>
      <td>0.101300</td>
      <td>0.311400</td>
      <td>0.104700</td>
      <td>0.279500</td>
      <td>0.051600</td>
      <td>0.338500</td>
      <td>0.173000</td>
      <td>0.353300</td>
    </tr>
    <tr>
      <td>11</td>
      <td>1.599700</td>
      <td>1.528889</td>
      <td>0.192200</td>
      <td>0.415000</td>
      <td>0.160800</td>
      <td>0.053700</td>
      <td>0.150500</td>
      <td>0.378000</td>
      <td>0.211500</td>
      <td>0.371700</td>
      <td>0.397800</td>
      <td>0.204900</td>
      <td>0.374600</td>
      <td>0.684800</td>
      <td>0.491900</td>
      <td>0.632400</td>
      <td>0.131200</td>
      <td>0.346800</td>
      <td>0.122000</td>
      <td>0.300900</td>
      <td>0.038400</td>
      <td>0.344600</td>
      <td>0.177500</td>
      <td>0.364400</td>
    </tr>
    <tr>
      <td>12</td>
      <td>1.599700</td>
      <td>1.517532</td>
      <td>0.198300</td>
      <td>0.429800</td>
      <td>0.159800</td>
      <td>0.066400</td>
      <td>0.162900</td>
      <td>0.383300</td>
      <td>0.220700</td>
      <td>0.382100</td>
      <td>0.405400</td>
      <td>0.214800</td>
      <td>0.383200</td>
      <td>0.672900</td>
      <td>0.469000</td>
      <td>0.610400</td>
      <td>0.167800</td>
      <td>0.379700</td>
      <td>0.119700</td>
      <td>0.307100</td>
      <td>0.038100</td>
      <td>0.335400</td>
      <td>0.196800</td>
      <td>0.394200</td>
    </tr>
    <tr>
      <td>13</td>
      <td>1.599700</td>
      <td>1.488849</td>
      <td>0.209800</td>
      <td>0.452300</td>
      <td>0.172300</td>
      <td>0.094900</td>
      <td>0.171100</td>
      <td>0.437800</td>
      <td>0.222000</td>
      <td>0.


key: _node_content
string_value:
{"id_": "50d2eb20-db81-4a6d-b7f6-bdcd382ffd8e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9a62800a-722d-42bf-b3ee-b8fe518b9f8e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "fa6a0a5dbcfef7e6c887051b1af0e5d3b5623fe6a590604f18fb73b741a466a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "67746dec-1552-40ab-8b3f-6cf676008054", "node_type": "1", "metadata": {}, "hash": "731649e8180d4142118bf4b3b7856cdf3678f3dab28ef85d29b22d16af794ae7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32362, "end_char_idx": 34068, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
599700</td>
      <td>1.517532</td>
      <td>0.198300</td>
      <td>0.429800</td>
      <td>0.159800</td>
      <td>0.066400</td>
      <td>0.162900</td>
      <td>0.383300</td>
      <td>0.220700</td>
      <td>0.382100</td>
      <td>0.405400</td>
      <td>0.214800</td>
      <td>0.383200</td>
      <td>0.672900</td>
      <td>0.469000</td>
      <td>0.610400</td>
      <td>0.167800</td>
      <td>0.379700</td>
      <td>0.119700</td>
      <td>0.307100</td>
      <td>0.038100</td>
      <td>0.335400</td>
      <td>0.196800</td>
      <td>0.394200</td>
    </tr>
    <tr>
      <td>13</td>
      <td>1.599700</td>
      <td>1.488849</td>
      <td>0.209800</td>
      <td>0.452300</td>
      <td>0.172300</td>
      <td>0.094900</td>
      <td>0.171100</td>
      <td>0.437800</td>
      <td>0.222000</td>
      <td>0.379800</td>
      <td>0.411500</td>
      <td>0.203800</td>
      <td>0.397300</td>
      <td>0.707500</td>
      <td>0.470700</td>
      <td>0.620700</td>
      <td>0.186900</td>
      <td>0.407600</td>
      <td>0.124200</td>
      <td>0.306700</td>
      <td>0.059300</td>
      <td>0.355400</td>
      <td>0.207700</td>
      <td>0.367100</td>
    </tr>
    <tr>
      <td>14</td>
      <td>1.599700</td>
      <td>1.482210</td>
      <td>0.228900</td>
      <td>0.482600</td>
      <td>0.187800</td>
      <td>0.083600</td>
      <td>0.191800</td>
      <td>0.444100</td>
      <td>0.225900</td>
      <td>0.376900</td>
      <td>0.407400</td>
      <td>0.182500</td>
      <td>0.384800</td>
      <td>0.700600</td>
      <td>0.512100</td>
      <td>0.640100</td>
      <td>0.175000</td>
      <td>0.363300</td>
      <td>0.144300</td>
      <td>0.300000</td>
      <td>0.


key: _node_content
string_value:
{"id_": "67746dec-1552-40ab-8b3f-6cf676008054", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50d2eb20-db81-4a6d-b7f6-bdcd382ffd8e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "c3305f0444a95b1ce062746d93e630bdbeb9578c61faac67b69331eec035affd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37e4a0c2-090c-4ca9-9f00-29ac1ba49a0e", "node_type": "1", "metadata": {}, "hash": "bcb0030986e11960b288b388d7a50c4f69aa5ec68021c30024638e43758bd823", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33215, "end_char_idx": 34910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
411500</td>
      <td>0.203800</td>
      <td>0.397300</td>
      <td>0.707500</td>
      <td>0.470700</td>
      <td>0.620700</td>
      <td>0.186900</td>
      <td>0.407600</td>
      <td>0.124200</td>
      <td>0.306700</td>
      <td>0.059300</td>
      <td>0.355400</td>
      <td>0.207700</td>
      <td>0.367100</td>
    </tr>
    <tr>
      <td>14</td>
      <td>1.599700</td>
      <td>1.482210</td>
      <td>0.228900</td>
      <td>0.482600</td>
      <td>0.187800</td>
      <td>0.083600</td>
      <td>0.191800</td>
      <td>0.444100</td>
      <td>0.225900</td>
      <td>0.376900</td>
      <td>0.407400</td>
      <td>0.182500</td>
      <td>0.384800</td>
      <td>0.700600</td>
      <td>0.512100</td>
      <td>0.640100</td>
      <td>0.175000</td>
      <td>0.363300</td>
      <td>0.144300</td>
      <td>0.300000</td>
      <td>0.083100</td>
      <td>0.363100</td>
      <td>0.229900</td>
      <td>0.370700</td>
    </tr>
    <tr>
      <td>15</td>
      <td>1.326800</td>
      <td>1.475198</td>
      <td>0.216300</td>
      <td>0.455600</td>
      <td>0.174900</td>
      <td>0.088500</td>
      <td>0.183500</td>
      <td>0.424400</td>
      <td>0.226900</td>
      <td>0.373400</td>
      <td>0.404300</td>
      <td>0.199200</td>
      <td>0.396400</td>
      <td>0.677800</td>
      <td>0.496300</td>
      <td>0.633800</td>
      <td>0.166300</td>
      <td>0.392400</td>
      <td>0.128900</td>
      <td>0.312900</td>
      <td>0.085200</td>
      <td>0.312300</td>
      <td>0.205000</td>
      <td>0.370200</td>
    </tr>
    <tr>
      <td>16</td>
      <td>1.326800</td>
      <td>1.459697</td>
      <td>0.233200</td>
      <td>0.504200</td>
      <td>0.


key: _node_content
string_value:
{"id_": "37e4a0c2-090c-4ca9-9f00-29ac1ba49a0e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "67746dec-1552-40ab-8b3f-6cf676008054", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "1ba39246a21a92b4e93f14743a981bd4f196bb91ea807c34ed4c15619f00d91c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "316081b7-d21a-4d9b-b216-813d20c82ad4", "node_type": "1", "metadata": {}, "hash": "40aa5e12de6a32ecd22d4f44d8e0ab074afa3fd921ef7e13c8d13e03ce92f3fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34044, "end_char_idx": 35739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
300000</td>
      <td>0.083100</td>
      <td>0.363100</td>
      <td>0.229900</td>
      <td>0.370700</td>
    </tr>
    <tr>
      <td>15</td>
      <td>1.326800</td>
      <td>1.475198</td>
      <td>0.216300</td>
      <td>0.455600</td>
      <td>0.174900</td>
      <td>0.088500</td>
      <td>0.183500</td>
      <td>0.424400</td>
      <td>0.226900</td>
      <td>0.373400</td>
      <td>0.404300</td>
      <td>0.199200</td>
      <td>0.396400</td>
      <td>0.677800</td>
      <td>0.496300</td>
      <td>0.633800</td>
      <td>0.166300</td>
      <td>0.392400</td>
      <td>0.128900</td>
      <td>0.312900</td>
      <td>0.085200</td>
      <td>0.312300</td>
      <td>0.205000</td>
      <td>0.370200</td>
    </tr>
    <tr>
      <td>16</td>
      <td>1.326800</td>
      <td>1.459697</td>
      <td>0.233200</td>
      <td>0.504200</td>
      <td>0.192200</td>
      <td>0.096000</td>
      <td>0.202000</td>
      <td>0.430800</td>
      <td>0.239100</td>
      <td>0.382400</td>
      <td>0.412600</td>
      <td>0.219500</td>
      <td>0.403100</td>
      <td>0.670400</td>
      <td>0.485200</td>
      <td>0.625200</td>
      <td>0.196500</td>
      <td>0.410100</td>
      <td>0.135700</td>
      <td>0.299600</td>
      <td>0.123100</td>
      <td>0.356900</td>
      <td>0.225300</td>
      <td>0.371100</td>
    </tr>
    <tr>
      <td>17</td>
      <td>1.326800</td>
      <td>1.407340</td>
      <td>0.243400</td>
      <td>0.511900</td>
      <td>0.204500</td>
      <td>0.121000</td>
      <td>0.215700</td>
      <td>0.468000</td>
      <td>0.246200</td>
      <td>0.394600</td>
      <td>0.424200</td>
      <td>0.225900</td>
      <td>0.416100</td>
      <td>0.


key: _node_content
string_value:
{"id_": "316081b7-d21a-4d9b-b216-813d20c82ad4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "37e4a0c2-090c-4ca9-9f00-29ac1ba49a0e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "dfc1e6ba06dd99ebae95ad120c764691fa7ac0dde280063ea6249e454c89a90d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8f35df88-e0e7-4c0c-952a-85154d989a1e", "node_type": "1", "metadata": {}, "hash": "60fc58b601e16dfe46d558e67088d6c0dc26658f0cef9dafd42ec40a297db2b0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34886, "end_char_idx": 36592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
504200</td>
      <td>0.192200</td>
      <td>0.096000</td>
      <td>0.202000</td>
      <td>0.430800</td>
      <td>0.239100</td>
      <td>0.382400</td>
      <td>0.412600</td>
      <td>0.219500</td>
      <td>0.403100</td>
      <td>0.670400</td>
      <td>0.485200</td>
      <td>0.625200</td>
      <td>0.196500</td>
      <td>0.410100</td>
      <td>0.135700</td>
      <td>0.299600</td>
      <td>0.123100</td>
      <td>0.356900</td>
      <td>0.225300</td>
      <td>0.371100</td>
    </tr>
    <tr>
      <td>17</td>
      <td>1.326800</td>
      <td>1.407340</td>
      <td>0.243400</td>
      <td>0.511900</td>
      <td>0.204500</td>
      <td>0.121000</td>
      <td>0.215700</td>
      <td>0.468000</td>
      <td>0.246200</td>
      <td>0.394600</td>
      <td>0.424200</td>
      <td>0.225900</td>
      <td>0.416100</td>
      <td>0.705200</td>
      <td>0.494900</td>
      <td>0.638300</td>
      <td>0.224900</td>
      <td>0.430400</td>
      <td>0.157200</td>
      <td>0.317900</td>
      <td>0.115700</td>
      <td>0.369200</td>
      <td>0.224200</td>
      <td>0.365300</td>
    </tr>
    <tr>
      <td>18</td>
      <td>1.326800</td>
      <td>1.419522</td>
      <td>0.245100</td>
      <td>0.521500</td>
      <td>0.210000</td>
      <td>0.116100</td>
      <td>0.211500</td>
      <td>0.489900</td>
      <td>0.255400</td>
      <td>0.391600</td>
      <td>0.419700</td>
      <td>0.198800</td>
      <td>0.421200</td>
      <td>0.701400</td>
      <td>0.501800</td>
      <td>0.634200</td>
      <td>0.226700</td>
      <td>0.410100</td>
      <td>0.154400</td>
      <td>0.321400</td>
      <td>0.105900</td>
      <td>0.352300</td>
      <td>0.236700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "8f35df88-e0e7-4c0c-952a-85154d989a1e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "316081b7-d21a-4d9b-b216-813d20c82ad4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "251c6a99bf7a6596fdef8736f0108d8d906efcb782e0112f40c4d5233d1a5f67", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86718ac1-74bd-4942-9b8d-a4da1795e541", "node_type": "1", "metadata": {}, "hash": "542b0da50062f554f23e896be66f4b56de157dddec87e3888e7aa55a9d78b461", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35739, "end_char_idx": 37434, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
705200</td>
      <td>0.494900</td>
      <td>0.638300</td>
      <td>0.224900</td>
      <td>0.430400</td>
      <td>0.157200</td>
      <td>0.317900</td>
      <td>0.115700</td>
      <td>0.369200</td>
      <td>0.224200</td>
      <td>0.365300</td>
    </tr>
    <tr>
      <td>18</td>
      <td>1.326800</td>
      <td>1.419522</td>
      <td>0.245100</td>
      <td>0.521500</td>
      <td>0.210000</td>
      <td>0.116100</td>
      <td>0.211500</td>
      <td>0.489900</td>
      <td>0.255400</td>
      <td>0.391600</td>
      <td>0.419700</td>
      <td>0.198800</td>
      <td>0.421200</td>
      <td>0.701400</td>
      <td>0.501800</td>
      <td>0.634200</td>
      <td>0.226700</td>
      <td>0.410100</td>
      <td>0.154400</td>
      <td>0.321400</td>
      <td>0.105900</td>
      <td>0.352300</td>
      <td>0.236700</td>
      <td>0.380400</td>
    </tr>
    <tr>
      <td>19</td>
      <td>1.158600</td>
      <td>1.398764</td>
      <td>0.253600</td>
      <td>0.519200</td>
      <td>0.213600</td>
      <td>0.135200</td>
      <td>0.207700</td>
      <td>0.491900</td>
      <td>0.257300</td>
      <td>0.397300</td>
      <td>0.428000</td>
      <td>0.241400</td>
      <td>0.401800</td>
      <td>0.703500</td>
      <td>0.509700</td>
      <td>0.631100</td>
      <td>0.236700</td>
      <td>0.441800</td>
      <td>0.155900</td>
      <td>0.330800</td>
      <td>0.128100</td>
      <td>0.352300</td>
      <td>0.237500</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>20</td>
      <td>1.158600</td>
      <td>1.390591</td>
      <td>0.248800</td>
      <td>0.520200</td>
      <td>0.216600</td>
      <td>0.127500</td>
      <td>0.211400</td>
      <td>0.


key: _node_content
string_value:
{"id_": "86718ac1-74bd-4942-9b8d-a4da1795e541", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8f35df88-e0e7-4c0c-952a-85154d989a1e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "30bf00ad7851552349f0025fc4d1dcc2538e5287f66353a9c3c4e61e653a8eb5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35d86f44-2f48-4af2-b576-54185c98f661", "node_type": "1", "metadata": {}, "hash": "93b3ea1db122c98eaca065ba41f19e8e2324f6a83987cd488b573a47664bd632", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36568, "end_char_idx": 38263, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
236700</td>
      <td>0.380400</td>
    </tr>
    <tr>
      <td>19</td>
      <td>1.158600</td>
      <td>1.398764</td>
      <td>0.253600</td>
      <td>0.519200</td>
      <td>0.213600</td>
      <td>0.135200</td>
      <td>0.207700</td>
      <td>0.491900</td>
      <td>0.257300</td>
      <td>0.397300</td>
      <td>0.428000</td>
      <td>0.241400</td>
      <td>0.401800</td>
      <td>0.703500</td>
      <td>0.509700</td>
      <td>0.631100</td>
      <td>0.236700</td>
      <td>0.441800</td>
      <td>0.155900</td>
      <td>0.330800</td>
      <td>0.128100</td>
      <td>0.352300</td>
      <td>0.237500</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>20</td>
      <td>1.158600</td>
      <td>1.390591</td>
      <td>0.248800</td>
      <td>0.520200</td>
      <td>0.216600</td>
      <td>0.127500</td>
      <td>0.211400</td>
      <td>0.471900</td>
      <td>0.258300</td>
      <td>0.407000</td>
      <td>0.429100</td>
      <td>0.240300</td>
      <td>0.407600</td>
      <td>0.708500</td>
      <td>0.505800</td>
      <td>0.623400</td>
      <td>0.235500</td>
      <td>0.431600</td>
      <td>0.150000</td>
      <td>0.325000</td>
      <td>0.125700</td>
      <td>0.375400</td>
      <td>0.227200</td>
      <td>0.390200</td>
    </tr>
    <tr>
      <td>21</td>
      <td>1.158600</td>
      <td>1.360608</td>
      <td>0.262700</td>
      <td>0.544800</td>
      <td>0.222100</td>
      <td>0.134700</td>
      <td>0.230000</td>
      <td>0.487500</td>
      <td>0.269500</td>
      <td>0.413300</td>
      <td>0.436300</td>
      <td>0.236200</td>
      <td>0.419100</td>
      <td>0.709300</td>
      <td>0.514100</td>
      <td>0.637400</td>
      <td>0.


key: _node_content
string_value:
{"id_": "35d86f44-2f48-4af2-b576-54185c98f661", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86718ac1-74bd-4942-9b8d-a4da1795e541", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "951bb1a0313d9abcc0c2ebff3cd013338567684066044653cd4daf1544529aa3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60d7418e-6c6d-448a-92f0-5b93f6c78576", "node_type": "1", "metadata": {}, "hash": "57af6cbed4da502a79e4b24269be0a483bcb53320638af54749498360c69e9bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37410, "end_char_idx": 39105, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
211400</td>
      <td>0.471900</td>
      <td>0.258300</td>
      <td>0.407000</td>
      <td>0.429100</td>
      <td>0.240300</td>
      <td>0.407600</td>
      <td>0.708500</td>
      <td>0.505800</td>
      <td>0.623400</td>
      <td>0.235500</td>
      <td>0.431600</td>
      <td>0.150000</td>
      <td>0.325000</td>
      <td>0.125700</td>
      <td>0.375400</td>
      <td>0.227200</td>
      <td>0.390200</td>
    </tr>
    <tr>
      <td>21</td>
      <td>1.158600</td>
      <td>1.360608</td>
      <td>0.262700</td>
      <td>0.544800</td>
      <td>0.222100</td>
      <td>0.134700</td>
      <td>0.230000</td>
      <td>0.487500</td>
      <td>0.269500</td>
      <td>0.413300</td>
      <td>0.436300</td>
      <td>0.236200</td>
      <td>0.419100</td>
      <td>0.709300</td>
      <td>0.514100</td>
      <td>0.637400</td>
      <td>0.257200</td>
      <td>0.450600</td>
      <td>0.165100</td>
      <td>0.338400</td>
      <td>0.139400</td>
      <td>0.372300</td>
      <td>0.237700</td>
      <td>0.382700</td>
    </tr>
    <tr>
      <td>22</td>
      <td>1.158600</td>
      <td>1.368296</td>
      <td>0.262800</td>
      <td>0.542400</td>
      <td>0.236400</td>
      <td>0.137400</td>
      <td>0.228100</td>
      <td>0.498500</td>
      <td>0.266500</td>
      <td>0.409000</td>
      <td>0.433000</td>
      <td>0.239900</td>
      <td>0.418500</td>
      <td>0.697500</td>
      <td>0.520500</td>
      <td>0.641000</td>
      <td>0.257500</td>
      <td>0.455700</td>
      <td>0.162600</td>
      <td>0.334800</td>
      <td>0.140200</td>
      <td>0.353800</td>
      <td>0.233200</td>
      <td>0.379600</td>
    </tr>
    <tr>
      <td>23</td>
      <td>1.


key: _node_content
string_value:
{"id_": "60d7418e-6c6d-448a-92f0-5b93f6c78576", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35d86f44-2f48-4af2-b576-54185c98f661", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8a5e50d2e4cc9fa1dc7247d6454128228d51bceb81655fcaac2ad0891b50380", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c03d71de-6243-41e2-900a-7fc57f2a2f21", "node_type": "1", "metadata": {}, "hash": "996ab9ab2511bfe89af3d75724392bbcc4c404e57ef764587f1bad8474cba4d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38239, "end_char_idx": 39934, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
637400</td>
      <td>0.257200</td>
      <td>0.450600</td>
      <td>0.165100</td>
      <td>0.338400</td>
      <td>0.139400</td>
      <td>0.372300</td>
      <td>0.237700</td>
      <td>0.382700</td>
    </tr>
    <tr>
      <td>22</td>
      <td>1.158600</td>
      <td>1.368296</td>
      <td>0.262800</td>
      <td>0.542400</td>
      <td>0.236400</td>
      <td>0.137400</td>
      <td>0.228100</td>
      <td>0.498500</td>
      <td>0.266500</td>
      <td>0.409000</td>
      <td>0.433000</td>
      <td>0.239900</td>
      <td>0.418500</td>
      <td>0.697500</td>
      <td>0.520500</td>
      <td>0.641000</td>
      <td>0.257500</td>
      <td>0.455700</td>
      <td>0.162600</td>
      <td>0.334800</td>
      <td>0.140200</td>
      <td>0.353800</td>
      <td>0.233200</td>
      <td>0.379600</td>
    </tr>
    <tr>
      <td>23</td>
      <td>1.158600</td>
      <td>1.368176</td>
      <td>0.264800</td>
      <td>0.541100</td>
      <td>0.233100</td>
      <td>0.138200</td>
      <td>0.223900</td>
      <td>0.498700</td>
      <td>0.272300</td>
      <td>0.407400</td>
      <td>0.434400</td>
      <td>0.233100</td>
      <td>0.418300</td>
      <td>0.702000</td>
      <td>0.524400</td>
      <td>0.642300</td>
      <td>0.262300</td>
      <td>0.444300</td>
      <td>0.159700</td>
      <td>0.335300</td>
      <td>0.140500</td>
      <td>0.366200</td>
      <td>0.236900</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>24</td>
      <td>1.049700</td>
      <td>1.355271</td>
      <td>0.269700</td>
      <td>0.549200</td>
      <td>0.239100</td>
      <td>0.134700</td>
      <td>0.229900</td>
      <td>0.519200</td>
      <td>0.274800</td>
      <td>0.


key: _node_content
string_value:
{"id_": "c03d71de-6243-41e2-900a-7fc57f2a2f21", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60d7418e-6c6d-448a-92f0-5b93f6c78576", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "2274710aca1900ce7bd61489e15736ff04f1dee3267555670853ee6944022437", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7602d741-f5f1-4d84-9506-491781e3d02c", "node_type": "1", "metadata": {}, "hash": "dbe838976ba9fc5496fb92c7f0ae4aa3f4ae7689f8202c813028b5963526e33e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39105, "end_char_idx": 40811, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
158600</td>
      <td>1.368176</td>
      <td>0.264800</td>
      <td>0.541100</td>
      <td>0.233100</td>
      <td>0.138200</td>
      <td>0.223900</td>
      <td>0.498700</td>
      <td>0.272300</td>
      <td>0.407400</td>
      <td>0.434400</td>
      <td>0.233100</td>
      <td>0.418300</td>
      <td>0.702000</td>
      <td>0.524400</td>
      <td>0.642300</td>
      <td>0.262300</td>
      <td>0.444300</td>
      <td>0.159700</td>
      <td>0.335300</td>
      <td>0.140500</td>
      <td>0.366200</td>
      <td>0.236900</td>
      <td>0.384000</td>
    </tr>
    <tr>
      <td>24</td>
      <td>1.049700</td>
      <td>1.355271</td>
      <td>0.269700</td>
      <td>0.549200</td>
      <td>0.239100</td>
      <td>0.134700</td>
      <td>0.229900</td>
      <td>0.519200</td>
      <td>0.274800</td>
      <td>0.412700</td>
      <td>0.437600</td>
      <td>0.245400</td>
      <td>0.417200</td>
      <td>0.711200</td>
      <td>0.523200</td>
      <td>0.644100</td>
      <td>0.272100</td>
      <td>0.440500</td>
      <td>0.166700</td>
      <td>0.341500</td>
      <td>0.137700</td>
      <td>0.373800</td>
      <td>0.249000</td>
      <td>0.388000</td>
    </tr>
    <tr>
      <td>25</td>
      <td>1.049700</td>
      <td>1.355180</td>
      <td>0.272500</td>
      <td>0.547900</td>
      <td>0.243800</td>
      <td>0.149700</td>
      <td>0.229900</td>
      <td>0.523100</td>
      <td>0.272500</td>
      <td>0.415700</td>
      <td>0.442200</td>
      <td>0.256200</td>
      <td>0.420200</td>
      <td>0.705800</td>
      <td>0.523900</td>
      <td>0.639600</td>
      <td>0.271700</td>
      <td>0.451900</td>
      <td>0.166300</td>
      <td>0.346900</td>
      <td>0.


key: _node_content
string_value:
{"id_": "7602d741-f5f1-4d84-9506-491781e3d02c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c03d71de-6243-41e2-900a-7fc57f2a2f21", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "ce56cbeadc7786b08e6426b809999267703aa983e0a95a43a2a616f72eaccdf2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8f15e391-ed0f-4e81-9765-0a663e4d16d9", "node_type": "1", "metadata": {}, "hash": "3abbe9e02d0a0d8293915accdaf866ce7ad8f01ddb462d5df8340245582b53b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39958, "end_char_idx": 41653, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
437600</td>
      <td>0.245400</td>
      <td>0.417200</td>
      <td>0.711200</td>
      <td>0.523200</td>
      <td>0.644100</td>
      <td>0.272100</td>
      <td>0.440500</td>
      <td>0.166700</td>
      <td>0.341500</td>
      <td>0.137700</td>
      <td>0.373800</td>
      <td>0.249000</td>
      <td>0.388000</td>
    </tr>
    <tr>
      <td>25</td>
      <td>1.049700</td>
      <td>1.355180</td>
      <td>0.272500</td>
      <td>0.547900</td>
      <td>0.243800</td>
      <td>0.149700</td>
      <td>0.229900</td>
      <td>0.523100</td>
      <td>0.272500</td>
      <td>0.415700</td>
      <td>0.442200</td>
      <td>0.256200</td>
      <td>0.420200</td>
      <td>0.705800</td>
      <td>0.523900</td>
      <td>0.639600</td>
      <td>0.271700</td>
      <td>0.451900</td>
      <td>0.166300</td>
      <td>0.346900</td>
      <td>0.153700</td>
      <td>0.383100</td>
      <td>0.247000</td>
      <td>0.389300</td>
    </tr>
    <tr>
      <td>26</td>
      <td>1.049700</td>
      <td>1.349337</td>
      <td>0.275600</td>
      <td>0.556300</td>
      <td>0.246400</td>
      <td>0.146700</td>
      <td>0.234800</td>
      <td>0.516300</td>
      <td>0.274200</td>
      <td>0.418300</td>
      <td>0.440900</td>
      <td>0.248700</td>
      <td>0.418900</td>
      <td>0.705800</td>
      <td>0.523200</td>
      <td>0.636500</td>
      <td>0.274700</td>
      <td>0.440500</td>
      <td>0.172400</td>
      <td>0.349100</td>
      <td>0.155600</td>
      <td>0.384600</td>
      <td>0.252300</td>
      <td>0.393800</td>
    </tr>
    <tr>
      <td>27</td>
      <td>1.049700</td>
      <td>1.350782</td>
      <td>0.275200</td>
      <td>0.548700</td>
      <td>0.


key: _node_content
string_value:
{"id_": "8f15e391-ed0f-4e81-9765-0a663e4d16d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7602d741-f5f1-4d84-9506-491781e3d02c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "67c53f3690f2a7266fed0f38bf72b253a7dce1e6ec4a4ee3854fa3f3e3f606b8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "894c5308-3591-4626-9f7b-f6d3dea92858", "node_type": "1", "metadata": {}, "hash": "4582398db0e3677e90fa61794572820bde94caffadff190b79d14743f5f6c763", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40787, "end_char_idx": 42482, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
346900</td>
      <td>0.153700</td>
      <td>0.383100</td>
      <td>0.247000</td>
      <td>0.389300</td>
    </tr>
    <tr>
      <td>26</td>
      <td>1.049700</td>
      <td>1.349337</td>
      <td>0.275600</td>
      <td>0.556300</td>
      <td>0.246400</td>
      <td>0.146700</td>
      <td>0.234800</td>
      <td>0.516300</td>
      <td>0.274200</td>
      <td>0.418300</td>
      <td>0.440900</td>
      <td>0.248700</td>
      <td>0.418900</td>
      <td>0.705800</td>
      <td>0.523200</td>
      <td>0.636500</td>
      <td>0.274700</td>
      <td>0.440500</td>
      <td>0.172400</td>
      <td>0.349100</td>
      <td>0.155600</td>
      <td>0.384600</td>
      <td>0.252300</td>
      <td>0.393800</td>
    </tr>
    <tr>
      <td>27</td>
      <td>1.049700</td>
      <td>1.350782</td>
      <td>0.275200</td>
      <td>0.548700</td>
      <td>0.246800</td>
      <td>0.147300</td>
      <td>0.236400</td>
      <td>0.527200</td>
      <td>0.280100</td>
      <td>0.416200</td>
      <td>0.442600</td>
      <td>0.253400</td>
      <td>0.424000</td>
      <td>0.710300</td>
      <td>0.526600</td>
      <td>0.640100</td>
      <td>0.273200</td>
      <td>0.445600</td>
      <td>0.167000</td>
      <td>0.346900</td>
      <td>0.160100</td>
      <td>0.387700</td>
      <td>0.249200</td>
      <td>0.392900</td>
    </tr>
    <tr>
      <td>28</td>
      <td>1.049700</td>
      <td>1.346533</td>
      <td>0.277000</td>
      <td>0.552800</td>
      <td>0.252900</td>
      <td>0.147400</td>
      <td>0.240000</td>
      <td>0.527600</td>
      <td>0.280900</td>
      <td>0.420900</td>
      <td>0.444100</td>
      <td>0.255500</td>
      <td>0.424500</td>
      <td>0.


key: _node_content
string_value:
{"id_": "894c5308-3591-4626-9f7b-f6d3dea92858", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8f15e391-ed0f-4e81-9765-0a663e4d16d9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "71a4dfeb97be5e0dd1945d4f52692e30e792b6c12e76525ddf9313ee358ad231", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a62196d-f409-40d7-8475-00972ac751b7", "node_type": "1", "metadata": {}, "hash": "0d37487fe2ddc2dac540596c43bf0cd3cc9a2c8b8dcedea6c92bf5d293765054", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41629, "end_char_idx": 43335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
548700</td>
      <td>0.246800</td>
      <td>0.147300</td>
      <td>0.236400</td>
      <td>0.527200</td>
      <td>0.280100</td>
      <td>0.416200</td>
      <td>0.442600</td>
      <td>0.253400</td>
      <td>0.424000</td>
      <td>0.710300</td>
      <td>0.526600</td>
      <td>0.640100</td>
      <td>0.273200</td>
      <td>0.445600</td>
      <td>0.167000</td>
      <td>0.346900</td>
      <td>0.160100</td>
      <td>0.387700</td>
      <td>0.249200</td>
      <td>0.392900</td>
    </tr>
    <tr>
      <td>28</td>
      <td>1.049700</td>
      <td>1.346533</td>
      <td>0.277000</td>
      <td>0.552800</td>
      <td>0.252900</td>
      <td>0.147400</td>
      <td>0.240000</td>
      <td>0.527600</td>
      <td>0.280900</td>
      <td>0.420900</td>
      <td>0.444100</td>
      <td>0.255500</td>
      <td>0.424500</td>
      <td>0.711200</td>
      <td>0.530200</td>
      <td>0.646800</td>
      <td>0.277400</td>
      <td>0.441800</td>
      <td>0.170900</td>
      <td>0.346900</td>
      <td>0.156600</td>
      <td>0.389200</td>
      <td>0.249600</td>
      <td>0.396000</td>
    </tr>
    <tr>
      <td>29</td>
      <td>0.993700</td>
      <td>1.346575</td>
      <td>0.277100</td>
      <td>0.554800</td>
      <td>0.252900</td>
      <td>0.148400</td>
      <td>0.239700</td>
      <td>0.523600</td>
      <td>0.278400</td>
      <td>0.420000</td>
      <td>0.443300</td>
      <td>0.256300</td>
      <td>0.424000</td>
      <td>0.705600</td>
      <td>0.529600</td>
      <td>0.647300</td>
      <td>0.273900</td>
      <td>0.439200</td>
      <td>0.174300</td>
      <td>0.348700</td>
      <td>0.157600</td>
      <td>0.386200</td>
      <td>0.250100</td>
      <td>0.


key: _node_content
string_value:
{"id_": "4a62196d-f409-40d7-8475-00972ac751b7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "894c5308-3591-4626-9f7b-f6d3dea92858", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "db7f158d38c6c0059eebe5b6968df44a3321b11cef9e692c4c210250307596ea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "acb8f6a3-41f0-47fb-9465-6562457240d4", "node_type": "1", "metadata": {}, "hash": "1ef026ef0d3a9ed9437da9b2d6b1a80a19cc4383f0da115e2f1842ad9d53c285", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42482, "end_char_idx": 44245, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
711200</td>
      <td>0.530200</td>
      <td>0.646800</td>
      <td>0.277400</td>
      <td>0.441800</td>
      <td>0.170900</td>
      <td>0.346900</td>
      <td>0.156600</td>
      <td>0.389200</td>
      <td>0.249600</td>
      <td>0.396000</td>
    </tr>
    <tr>
      <td>29</td>
      <td>0.993700</td>
      <td>1.346575</td>
      <td>0.277100</td>
      <td>0.554800</td>
      <td>0.252900</td>
      <td>0.148400</td>
      <td>0.239700</td>
      <td>0.523600</td>
      <td>0.278400</td>
      <td>0.420000</td>
      <td>0.443300</td>
      <td>0.256300</td>
      <td>0.424000</td>
      <td>0.705600</td>
      <td>0.529600</td>
      <td>0.647300</td>
      <td>0.273900</td>
      <td>0.439200</td>
      <td>0.174300</td>
      <td>0.348700</td>
      <td>0.157600</td>
      <td>0.386200</td>
      <td>0.250100</td>
      <td>0.395100</td>
    </tr>
    <tr>
      <td>30</td>
      <td>0.993700</td>
      <td>1.346446</td>
      <td>0.277400</td>
      <td>0.554700</td>
      <td>0.252700</td>
      <td>0.147900</td>
      <td>0.240800</td>
      <td>0.523600</td>
      <td>0.278800</td>
      <td>0.420400</td>
      <td>0.443300</td>
      <td>0.256100</td>
      <td>0.424200</td>
      <td>0.705500</td>
      <td>0.530100</td>
      <td>0.646800</td>
      <td>0.275600</td>
      <td>0.440500</td>
      <td>0.174500</td>
      <td>0.348700</td>
      <td>0.157300</td>
      <td>0.386200</td>
      <td>0.249200</td>
      <td>0.394200</td>
    </tr>
  </tbody>
</table><p>

If you have set `push_to_hub` to `True` in the `training_args`, the training checkpoints are pushed to the
Hugging Face Hub. Upon training completion, push the final model to the Hub as well by calling the [`~transformers.Trainer.push_to_hub`] method.


key: _node_content
string_value:
{"id_": "acb8f6a3-41f0-47fb-9465-6562457240d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a62196d-f409-40d7-8475-00972ac751b7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "5ee60d7662f6e338b0b5f8bd0b3c28daf245c536acf592986919ac276630b9f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c946fdf-6bf4-4ecf-87f5-f85476425e1c", "node_type": "1", "metadata": {}, "hash": "fef76698782cd924dfa6d083953b9935fd0dc238a9ff8043046ec52647f20817", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43263, "end_char_idx": 45324, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
157600</td>
      <td>0.386200</td>
      <td>0.250100</td>
      <td>0.395100</td>
    </tr>
    <tr>
      <td>30</td>
      <td>0.993700</td>
      <td>1.346446</td>
      <td>0.277400</td>
      <td>0.554700</td>
      <td>0.252700</td>
      <td>0.147900</td>
      <td>0.240800</td>
      <td>0.523600</td>
      <td>0.278800</td>
      <td>0.420400</td>
      <td>0.443300</td>
      <td>0.256100</td>
      <td>0.424200</td>
      <td>0.705500</td>
      <td>0.530100</td>
      <td>0.646800</td>
      <td>0.275600</td>
      <td>0.440500</td>
      <td>0.174500</td>
      <td>0.348700</td>
      <td>0.157300</td>
      <td>0.386200</td>
      <td>0.249200</td>
      <td>0.394200</td>
    </tr>
  </tbody>
</table><p>

If you have set `push_to_hub` to `True` in the `training_args`, the training checkpoints are pushed to the
Hugging Face Hub. Upon training completion, push the final model to the Hub as well by calling the [`~transformers.Trainer.push_to_hub`] method.

```py
>>> trainer.push_to_hub()
```

## Evaluate

```py
>>> from pprint import pprint

>>> metrics = trainer.evaluate(eval_dataset=cppe5["test"], metric_key_prefix="test")
>>> pprint(metrics)
{'epoch': 30.0,
  'test_loss': 1.0877351760864258,
  'test_map': 0.4116,
  'test_map_50': 0.741,
  'test_map_75': 0.3663,
  'test_map_Coverall': 0.5937,
  'test_map_Face_Shield': 0.5863,
  'test_map_Gloves': 0.3416,
  'test_map_Goggles': 0.1468,
  'test_map_Mask': 0.3894,
  'test_map_large': 0.5637,
  'test_map_medium': 0.3257,
  'test_map_small': 0.3589,
  'test_mar_1': 0.323,
  'test_mar_10': 0.5237,
  'test_mar_100': 0.5587,
  'test_mar_100_Coverall': 0.6756,
  'test_mar_100_Face_Shield': 0.7294,
  'test_mar_100_Gloves': 0.4721,
  'test_mar_100_Goggles': 0.4125,
  'test_mar_100_Mask': 0.5038,
  'test_mar_large': 0.7283,
  'test_mar_medium': 0.4901,
  'test_mar_small': 0.4469,
  'test_runtime': 1.6526,
  'test_samples_per_second': 17.548,
  'test_steps_per_second': 2.42}
```

These results can be further improved by adjusting the hyperparameters in [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "2c946fdf-6bf4-4ecf-87f5-f85476425e1c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86d3632c-e406-4799-87f1-6d92f0bc08d3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "f8cde93ad52fd17098aae9306b38cc7ca21b5f3a1474eefb572ef818d732cebe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "acb8f6a3-41f0-47fb-9465-6562457240d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection/", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "HF_Transformers"}, "hash": "d5f9790d820b00351e698d8dc62eca39dbbf52497acd2e4811130521685100ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 45325, "end_char_idx": 47788, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Give it a go!

## Inference

Now that you have finetuned a model, evaluated it, and uploaded it to the Hugging Face Hub, you can use it for inference.

```py
>>> import torch
>>> import requests

>>> from PIL import Image, ImageDraw
>>> from transformers import AutoImageProcessor, AutoModelForObjectDetection

>>> url = "https://images.pexels.com/photos/8413299/pexels-photo-8413299.jpeg?auto=compress&cs=tinysrgb&w=630&h=375&dpr=2"
>>> image = Image.open(requests.get(url, stream=True).raw)
```

Load model and image processor from the Hugging Face Hub (skip to use already trained in this session):
```py
>>> device = "cuda"
>>> model_repo = "qubvel-hf/detr_finetuned_cppe5"

>>> image_processor = AutoImageProcessor.from_pretrained(model_repo)
>>> model = AutoModelForObjectDetection.from_pretrained(model_repo)
>>> model = model.to(device)
```

And detect bounding boxes:

```py

>>> with torch.no_grad():
...     inputs = image_processor(images=[image], return_tensors="pt")
...     outputs = model(**inputs.to(device))
...     target_sizes = torch.tensor([[image.size[1], image.size[0]]])
...     results = image_processor.post_process_object_detection(outputs, threshold=0.3, target_sizes=target_sizes)[0]

>>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
...     box = [round(i, 2) for i in box.tolist()]
...     print(
...         f"Detected {model.config.id2label[label.item()]} with confidence "
...         f"{round(score.item(), 3)} at location {box}"
...     )
Detected Gloves with confidence 0.683 at location [244.58, 124.33, 300.35, 185.13]
Detected Mask with confidence 0.517 at location [143.73, 64.58, 219.57, 125.89]
Detected Gloves with confidence 0.425 at location [179.15, 155.57, 262.4, 226.35]
Detected Coverall with confidence 0.407 at location [307.13, -1.18, 477.82, 318.06]
Detected Coverall with confidence 0.391 at location [68.61, 126.66, 309.03, 318.89]
```

Let's plot the result:

```py
>>> draw = ImageDraw.Draw(image)

>>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
...     box = [round(i, 2) for i in box.tolist()]
...     x, y, x2, y2 = tuple(box)
...     draw.rectangle((x, y, x2, y2), outline="red", width=1)
...     draw.text((x, y), model.config.id2label[label.item()], fill="white")

>>> image
```

<div class="flex justify-center">
    <img src="https://i.imgur.com/oDUqD0K.png" alt="Object detection result on a new image"/>
</div>


key: _node_content
string_value:
{"id_": "4a811231-2bc9-4ffb-8f47-732b80113300", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "67c0af2a-7fc7-41ec-bd5e-5d79a6984988", "node_type": "1", "metadata": {}, "hash": "7fad541e01f49fb3c14c95628ab35eaf9915dee7237b3c7b3e8f2e6bf913debd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2505, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Token classification

[[open-in-colab]]

<Youtube id="wVHdVlPScxA"/>

Token classification assigns a label to individual tokens in a sentence. One of the most common token classification tasks is Named Entity Recognition (NER). NER attempts to find a label for each entity in a sentence, such as a person, location, or organization.

This guide will show you how to:

1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [WNUT 17](https://huggingface.co/datasets/wnut_17) dataset to detect new entities.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/token-classification).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate seqeval
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load WNUT 17 dataset

Start by loading the WNUT 17 dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> wnut = load_dataset("wnut_17")
```

Then take a look at an example:

```py
>>> wnut["train"][0]
{'id': '0',
 'ner_tags': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 8, 8, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0],
 'tokens': ['@paulwalk', 'It', "'s", 'the', 'view', 'from', 'where', 'I', "'m", 'living', 'for', 'two', 'weeks', '.', 'Empire', 'State', 'Building', '=', 'ESB', '.', 'Pretty', 'bad', 'storm', 'here', 'last', 'evening', '.']
}
```

Each number in `ner_tags` represents an entity. Convert the numbers to their label names to find out what the entities are:

```py
>>> label_list = wnut["train"].features[f"ner_tags"].feature.names
>>> label_list
[
    "O",
    "B-corporation",
    "I-corporation",
    "B-creative-work",
    "I-creative-work",
    "B-group",
    "I-group",
    "B-location",
    "I-location",
    "B-person",
    "I-person",
    "B-product",
    "I-product",
]
```

The letter that prefixes each `ner_tag` indicates the token position of the entity:

- `B-` indicates the beginning of an entity.
- `I-` indicates a token is contained inside the same entity (for example, the `State` token is a part of an entity like
  `Empire State Building`).
- `0` indicates the token doesn't correspond to any entity.


key: _node_content
string_value:
{"id_": "67c0af2a-7fc7-41ec-bd5e-5d79a6984988", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a811231-2bc9-4ffb-8f47-732b80113300", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0fa17abb55d484c34b811fde5c916c493c0fb21241defa9d5b6598b51bf2ec1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82b0299d-aa18-4db6-9b34-c3167eff864c", "node_type": "1", "metadata": {}, "hash": "e24a3259e936aa3c545d1ab2fdad9f602f66c385b824ec4ef84154749c1c20b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1600, "end_char_idx": 4255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', 'Empire', 'State', 'Building', '=', 'ESB', '.', 'Pretty', 'bad', 'storm', 'here', 'last', 'evening', '.']
}
```

Each number in `ner_tags` represents an entity. Convert the numbers to their label names to find out what the entities are:

```py
>>> label_list = wnut["train"].features[f"ner_tags"].feature.names
>>> label_list
[
    "O",
    "B-corporation",
    "I-corporation",
    "B-creative-work",
    "I-creative-work",
    "B-group",
    "I-group",
    "B-location",
    "I-location",
    "B-person",
    "I-person",
    "B-product",
    "I-product",
]
```

The letter that prefixes each `ner_tag` indicates the token position of the entity:

- `B-` indicates the beginning of an entity.
- `I-` indicates a token is contained inside the same entity (for example, the `State` token is a part of an entity like
  `Empire State Building`).
- `0` indicates the token doesn't correspond to any entity.

## Preprocess

<Youtube id="iY2AZYdZAr0"/>

The next step is to load a DistilBERT tokenizer to preprocess the `tokens` field:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

As you saw in the example `tokens` field above, it looks like the input has already been tokenized. But the input actually hasn't been tokenized yet and you'll need to set `is_split_into_words=True` to tokenize the words into subwords. For example:

```py
>>> example = wnut["train"][0]
>>> tokenized_input = tokenizer(example["tokens"], is_split_into_words=True)
>>> tokens = tokenizer.convert_ids_to_tokens(tokenized_input["input_ids"])
>>> tokens
['[CLS]', '@', 'paul', '##walk', 'it', "'", 's', 'the', 'view', 'from', 'where', 'i', "'", 'm', 'living', 'for', 'two', 'weeks', '.', 'empire', 'state', 'building', '=', 'es', '##b', '.', 'pretty', 'bad', 'storm', 'here', 'last', 'evening', '.', '[SEP]']
```

However, this adds some special tokens `[CLS]` and `[SEP]` and the subword tokenization creates a mismatch between the input and labels. A single word corresponding to a single label may now be split into two subwords. You'll need to realign the tokens and labels by:

1. Mapping all tokens to their corresponding word with the [`word_ids`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.BatchEncoding.word_ids) method.
2. Assigning the label `-100` to the special tokens `[CLS]` and `[SEP]` so they're ignored by the PyTorch loss function (see [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)).
3. Only labeling the first token of a given word. Assign `-100` to other subtokens from the same word.


key: _node_content
string_value:
{"id_": "82b0299d-aa18-4db6-9b34-c3167eff864c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "67c0af2a-7fc7-41ec-bd5e-5d79a6984988", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d9d3ee0843ab18fed09e60d8a69b93fb57c7f3d2a799d311de7db302c585ad31", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b7ab219-0733-47c8-ad65-fa626a214fc9", "node_type": "1", "metadata": {}, "hash": "8b40dc6404aa43958b99d6dba236aab721b1780867b9fbaff0852f08e6ba03de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2873, "end_char_idx": 5883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But the input actually hasn't been tokenized yet and you'll need to set `is_split_into_words=True` to tokenize the words into subwords. For example:

```py
>>> example = wnut["train"][0]
>>> tokenized_input = tokenizer(example["tokens"], is_split_into_words=True)
>>> tokens = tokenizer.convert_ids_to_tokens(tokenized_input["input_ids"])
>>> tokens
['[CLS]', '@', 'paul', '##walk', 'it', "'", 's', 'the', 'view', 'from', 'where', 'i', "'", 'm', 'living', 'for', 'two', 'weeks', '.', 'empire', 'state', 'building', '=', 'es', '##b', '.', 'pretty', 'bad', 'storm', 'here', 'last', 'evening', '.', '[SEP]']
```

However, this adds some special tokens `[CLS]` and `[SEP]` and the subword tokenization creates a mismatch between the input and labels. A single word corresponding to a single label may now be split into two subwords. You'll need to realign the tokens and labels by:

1. Mapping all tokens to their corresponding word with the [`word_ids`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.BatchEncoding.word_ids) method.
2. Assigning the label `-100` to the special tokens `[CLS]` and `[SEP]` so they're ignored by the PyTorch loss function (see [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)).
3. Only labeling the first token of a given word. Assign `-100` to other subtokens from the same word.

Here is how you can create a function to realign the tokens and labels, and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def tokenize_and_align_labels(examples):
...     tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)

...     labels = []
...     for i, label in enumerate(examples[f"ner_tags"]):
...         word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
...         previous_word_idx = None
...         label_ids = []
...         for word_idx in word_ids:  # Set the special tokens to -100.
...             if word_idx is None:
...                 label_ids.append(-100)
...             elif word_idx != previous_word_idx:  # Only label the first token of a given word.
...                 label_ids.append(label[word_idx])
...             else:
...                 label_ids.append(-100)
...             previous_word_idx = word_idx
...         labels.append(label_ids)

...     tokenized_inputs["labels"] = labels
...     return tokenized_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.


key: _node_content
string_value:
{"id_": "8b7ab219-0733-47c8-ad65-fa626a214fc9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82b0299d-aa18-4db6-9b34-c3167eff864c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "314996315bfef29a5555fff77a218f63d80442dbb260a9427a02ceeadf8f61a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43148f67-acb3-4b9b-b66d-7e16ee19ca4a", "node_type": "1", "metadata": {}, "hash": "c48f8e218afc02521019a541ca338e311e5413d985d6935bb41b53308401feca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4153, "end_char_idx": 7140, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Only labeling the first token of a given word. Assign `-100` to other subtokens from the same word.

Here is how you can create a function to realign the tokens and labels, and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def tokenize_and_align_labels(examples):
...     tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)

...     labels = []
...     for i, label in enumerate(examples[f"ner_tags"]):
...         word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
...         previous_word_idx = None
...         label_ids = []
...         for word_idx in word_ids:  # Set the special tokens to -100.
...             if word_idx is None:
...                 label_ids.append(-100)
...             elif word_idx != previous_word_idx:  # Only label the first token of a given word.
...                 label_ids.append(label[word_idx])
...             else:
...                 label_ids.append(-100)
...             previous_word_idx = word_idx
...         labels.append(label_ids)

...     tokenized_inputs["labels"] = labels
...     return tokenized_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) framework (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric). Seqeval actually produces several scores: precision, recall, F1, and accuracy.

```py
>>> import evaluate

>>> seqeval = evaluate.load("seqeval")
```

Get the NER labels first, and then create a function that passes your true predictions and true labels to [`~evaluate.EvaluationModule.compute`] to calculate the scores:

```py
>>> import numpy as np

>>> labels = [label_list[i] for i in example[f"ner_tags"]]


key: _node_content
string_value:
{"id_": "43148f67-acb3-4b9b-b66d-7e16ee19ca4a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b7ab219-0733-47c8-ad65-fa626a214fc9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8874d972dc3a196ac586e93c3bbae2bea8a9326b5a968441e3455b66f932efb9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f05cf722-4bab-408b-ab51-cf731f62e031", "node_type": "1", "metadata": {}, "hash": "2785ed4cda887f25cbbfdbf7f11d1469fc8fb6cbc35dc7fd9c5fb56868f7f0ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5454, "end_char_idx": 8034, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) framework (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric). Seqeval actually produces several scores: precision, recall, F1, and accuracy.

```py
>>> import evaluate

>>> seqeval = evaluate.load("seqeval")
```

Get the NER labels first, and then create a function that passes your true predictions and true labels to [`~evaluate.EvaluationModule.compute`] to calculate the scores:

```py
>>> import numpy as np

>>> labels = [label_list[i] for i in example[f"ner_tags"]]


>>> def compute_metrics(p):
...     predictions, labels = p
...     predictions = np.argmax(predictions, axis=2)

...     true_predictions = [
...         [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]
...     true_labels = [
...         [label_list[l] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]

...     results = seqeval.compute(predictions=true_predictions, references=true_labels)
...     return {
...         "precision": results["overall_precision"],
...         "recall": results["overall_recall"],
...         "f1": results["overall_f1"],
...         "accuracy": results["overall_accuracy"],
...     }
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.


key: _node_content
string_value:
{"id_": "f05cf722-4bab-408b-ab51-cf731f62e031", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43148f67-acb3-4b9b-b66d-7e16ee19ca4a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c40cf9b27b8aad15a4ac6508ddfe7614709cabc46251049585e403aed6cb6e8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d5ad43c-b77d-4745-9837-56d454e73e3c", "node_type": "1", "metadata": {}, "hash": "361ddae4d20d8a9763a0df4fb87d8f3d209da7e2a76e44dc9857432d305f8b1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6376, "end_char_idx": 9106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) framework (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric). Seqeval actually produces several scores: precision, recall, F1, and accuracy.

```py
>>> import evaluate

>>> seqeval = evaluate.load("seqeval")
```

Get the NER labels first, and then create a function that passes your true predictions and true labels to [`~evaluate.EvaluationModule.compute`] to calculate the scores:

```py
>>> import numpy as np

>>> labels = [label_list[i] for i in example[f"ner_tags"]]


>>> def compute_metrics(p):
...     predictions, labels = p
...     predictions = np.argmax(predictions, axis=2)

...     true_predictions = [
...         [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]
...     true_labels = [
...         [label_list[l] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]

...     results = seqeval.compute(predictions=true_predictions, references=true_labels)
...     return {
...         "precision": results["overall_precision"],
...         "recall": results["overall_recall"],
...         "f1": results["overall_f1"],
...         "accuracy": results["overall_accuracy"],
...     }
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {
...     0: "O",
...     1: "B-corporation",
...     2: "I-corporation",
...     3: "B-creative-work",
...     4: "I-creative-work",
...     5: "B-group",
...     6: "I-group",
...     7: "B-location",
...     8: "I-location",
...     9: "B-person",
...     10: "I-person",
...     11: "B-product",
...     12: "I-product",
... }
>>> label2id = {
...     "O": 0,
...     "B-corporation": 1,
...     "I-corporation": 2,
...     "B-creative-work": 3,
...     "I-creative-work": 4,
...     "B-group": 5,
...     "I-group": 6,
...     "B-location": 7,
...     "I-location": 8,
...     "B-person": 9,
...     "I-person": 10,
...     "B-product": 11,
...     "I-product": 12,
... }
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now!


key: _node_content
string_value:
{"id_": "8d5ad43c-b77d-4745-9837-56d454e73e3c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f05cf722-4bab-408b-ab51-cf731f62e031", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e73078c8626bc13f399d96a98b9c09263a75654d702d129661443fe20743cbde", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b0f11b6-2961-4b4d-ab63-2fd4d4b247ca", "node_type": "1", "metadata": {}, "hash": "d14c6500bb75218c36c18dc1c412f9c3e9419f26650964188997e7522391b8ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8036, "end_char_idx": 10113, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {
...     0: "O",
...     1: "B-corporation",
...     2: "I-corporation",
...     3: "B-creative-work",
...     4: "I-creative-work",
...     5: "B-group",
...     6: "I-group",
...     7: "B-location",
...     8: "I-location",
...     9: "B-person",
...     10: "I-person",
...     11: "B-product",
...     12: "I-product",
... }
>>> label2id = {
...     "O": 0,
...     "B-corporation": 1,
...     "I-corporation": 2,
...     "B-creative-work": 3,
...     "I-creative-work": 4,
...     "B-group": 5,
...     "I-group": 6,
...     "B-location": 7,
...     "I-location": 8,
...     "B-person": 9,
...     "I-person": 10,
...     "B-product": 11,
...     "I-product": 12,
... }
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

>>> model = AutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the seqeval scores and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "7b0f11b6-2961-4b4d-ab63-2fd4d4b247ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d5ad43c-b77d-4745-9837-56d454e73e3c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5211c0b99e156a737819b9ca86b3c23426fd96305d174d418bc627a41fb9f8e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "384dcf27-b4a6-43d5-b233-52ac30656436", "node_type": "1", "metadata": {}, "hash": "884fdc423f1ee4a412f40282735107c2af27bc154c7ad28ac3141014acd96396", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9052, "end_char_idx": 11147, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

>>> model = AutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the seqeval scores and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_wnut_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_wnut["train"],
...     eval_dataset=tokenized_wnut["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "384dcf27-b4a6-43d5-b233-52ac30656436", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b0f11b6-2961-4b4d-ab63-2fd4d4b247ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6a84978ccc69c4fb68db235124568cb51788ed97b823dd028800947c95626f08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "834c5aef-5dcb-48c8-964e-6332a2de3fdb", "node_type": "1", "metadata": {}, "hash": "e02c7d35bf0351340539f9131f6e3d953747f6cec2f229514f303db3b2a22b5c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9604, "end_char_idx": 12620, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the seqeval scores and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_wnut_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_wnut["train"],
...     eval_dataset=tokenized_wnut["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 3
>>> num_train_steps = (len(tokenized_wnut["train"]) // batch_size) * num_train_epochs
>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=2e-5,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=0.01,
...     num_warmup_steps=0,
... )
```

Then you can load DistilBERT with [`TFAutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_wnut["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_wnut["validation"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method).


key: _node_content
string_value:
{"id_": "834c5aef-5dcb-48c8-964e-6332a2de3fdb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "384dcf27-b4a6-43d5-b233-52ac30656436", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "88152ea2fafbdccd4df75ca57cc4d2ed6a61013776fa08e79be8b6efbc74da75", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c2c66cb-92f9-4b8e-8a63-381a977e9fb5", "node_type": "1", "metadata": {}, "hash": "567448bec4c7d17bccd228fbf7f802103157af75729277c11b04a956639fbdb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11149, "end_char_idx": 14184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 3
>>> num_train_steps = (len(tokenized_wnut["train"]) // batch_size) * num_train_epochs
>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=2e-5,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=0.01,
...     num_warmup_steps=0,
... )
```

Then you can load DistilBERT with [`TFAutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_wnut["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_wnut["validation"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the seqeval scores from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_wnut_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!


key: _node_content
string_value:
{"id_": "7c2c66cb-92f9-4b8e-8a63-381a977e9fb5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "834c5aef-5dcb-48c8-964e-6332a2de3fdb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0186d62f959c097e506bbe3a5fcd9eda07fd8871914063a4c79febbddc935d92", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c19360b6-fb09-47bd-8c70-27493162f931", "node_type": "1", "metadata": {}, "hash": "43649f7ea7aa46f7b0026a055f23fdf5a41ddb27921fa2d7f9bbbb0eb0c58095", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12621, "end_char_idx": 15602, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the seqeval scores from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_wnut_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for token classification, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "The Golden State Warriors are an American professional basketball team based in San Francisco."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for NER with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("ner", model="stevhliu/my_awesome_wnut_model")
>>> classifier(text)
[{'entity': 'B-location',
  'score': 0.42658573,
  'index': 2,
  'word': 'golden',
  'start': 4,
  'end': 10},
 {'entity': 'I-location',
  'score': 0.35856336,
  'index': 3,
  'word': 'state',
  'start': 11,
  'end': 16},
 {'entity': 'B-group',
  'score': 0.3064001,
  'index': 4,
  'word': 'warriors',
  'start': 17,
  'end': 25},
 {'entity': 'B-location',
  'score': 0.65523505,
  'index': 13,
  'word': 'san',
  'start': 80,


key: _node_content
string_value:
{"id_": "c19360b6-fb09-47bd-8c70-27493162f931", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c2c66cb-92f9-4b8e-8a63-381a977e9fb5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2865ca0f33440d29a3d60503dd4011d3b4911ca45787c65e49df29fd4e0f242d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "faad2467-fee8-425d-8743-b089fa41827b", "node_type": "1", "metadata": {}, "hash": "e2b9c89db616ded8c9a9d885c31b8597cb656a91a069386591f1e5554b823a0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "The Golden State Warriors are an American professional basketball team based in San Francisco."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for NER with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("ner", model="stevhliu/my_awesome_wnut_model")
>>> classifier(text)
[{'entity': 'B-location',
  'score': 0.42658573,
  'index': 2,
  'word': 'golden',
  'start': 4,
  'end': 10},
 {'entity': 'I-location',
  'score': 0.35856336,
  'index': 3,
  'word': 'state',
  'start': 11,
  'end': 16},
 {'entity': 'B-group',
  'score': 0.3064001,
  'index': 4,
  'word': 'warriors',
  'start': 17,
  'end': 25},
 {'entity': 'B-location',
  'score': 0.65523505,
  'index': 13,
  'word': 'san',
  'start': 80,
  'end': 83},
 {'entity': 'B-location',
  'score': 0.4668663,
  'index': 14,
  'word': 'francisco',
  'start': 84,
  'end': 93}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> with torch.no_grad():
.     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predictions = torch.argmax(logits, dim=2)
>>> predicted_token_class = [model.config.id2label[t.item()] for t in predictions[0]]
>>> predicted_token_class
['O',
 'O',
 'B-location',
 'I-location',
 'B-group',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'B-location',
 'B-location',
 'O',
 'O']
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text,


key: _node_content
string_value:
{"id_": "faad2467-fee8-425d-8743-b089fa41827b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef179637-3fcd-4f56-995b-1535743660de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "896811ddc6e649ba0f279fade17340e056321961772472e72eb0e3b3463c3ba3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c19360b6-fb09-47bd-8c70-27493162f931", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification/", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "70c1822e68bfcc19bb9714ca891ed529b292401ab2eaf84af58b3a3025474f00", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4668663,
  'index': 14,
  'word': 'francisco',
  'start': 84,
  'end': 93}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> with torch.no_grad():
.     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predictions = torch.argmax(logits, dim=2)
>>> predicted_token_class = [model.config.id2label[t.item()] for t in predictions[0]]
>>> predicted_token_class
['O',
 'O',
 'B-location',
 'I-location',
 'B-group',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'B-location',
 'B-location',
 'O',
 'O']
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_token_class_ids = tf.math.argmax(logits, axis=-1)
>>> predicted_token_class = [model.config.id2label[t] for t in predicted_token_class_ids[0].numpy().tolist()]
>>> predicted_token_class
['O',
 'O',
 'B-location',
 'I-location',
 'B-group',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'B-location',
 'B-location',
 'O',
 'O']
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "544719e6-5e97-4abb-9342-78aaf21db192", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6e4ee3b-32bc-4552-b940-48096f7b5538", "node_type": "1", "metadata": {}, "hash": "c8f70e656c6b7ff5b593c66ace565599914eb6283fcc8a714c33a92590bfdcab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text classification

[[open-in-colab]]

<Youtube id="leNG9fN9FQU"/>

Text classification is a common NLP task that assigns a label or class to text. Some of the largest companies run text classification in production for a wide range of practical applications. One of the most popular forms of text classification is sentiment analysis, which assigns a label like 🙂 positive, 🙁 negative, or 😐 neutral to a sequence of text.

This guide will show you how to:

1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [IMDb](https://huggingface.co/datasets/imdb) dataset to determine whether a movie review is positive or negative.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/text-classification).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate accelerate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load IMDb dataset

Start by loading the IMDb dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> imdb = load_dataset("imdb")
```

Then take a look at an example:

```py
>>> imdb["test"][0]
{
    "label": 0,
    "text": "I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi' setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV. It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It's really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it's rubbish as they have to always say \"Gene Roddenberry's Earth...\" otherwise people would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.",
}
```

There are two fields in this dataset:

- `text`: the movie review text.
- `label`: a value that is either `0` for a negative review or `1` for a positive review.


key: _node_content
string_value:
{"id_": "a6e4ee3b-32bc-4552-b940-48096f7b5538", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "544719e6-5e97-4abb-9342-78aaf21db192", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "68f57b3515626f716eb4a5e10e130d49a3d527d7e62838225a7482d8ff97abe8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4969f29-c2a9-4b84-a27c-b5506a8ac4f8", "node_type": "1", "metadata": {}, "hash": "7ee99d2dff91ce91cd085ce898a4341ab4217ee25a1d9fbcfa28f5fc107c9594", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1573, "end_char_idx": 4641, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi' setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV. It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It's really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it's rubbish as they have to always say \"Gene Roddenberry's Earth...\" otherwise people would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.",
}
```

There are two fields in this dataset:

- `text`: the movie review text.
- `label`: a value that is either `0` for a negative review or `1` for a positive review.

## Preprocess

The next step is to load a DistilBERT tokenizer to preprocess the `text` field:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a preprocessing function to tokenize `text` and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def preprocess_function(examples):
...     return tokenizer(examples["text"], truncation=True)
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once:

```py
tokenized_imdb = imdb.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library.


key: _node_content
string_value:
{"id_": "c4969f29-c2a9-4b84-a27c-b5506a8ac4f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6e4ee3b-32bc-4552-b940-48096f7b5538", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1875e43c60ab256994a81e780315d606dd2ac5812c3f1ed3aca3e6b47da2fd70", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7baf32fb-7075-400a-a83e-8905e874067d", "node_type": "1", "metadata": {}, "hash": "b77769ff2fd9e32b51471f005094ed1162e8a4b968124abc65fbd7b02a882965", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2992, "end_char_idx": 5904, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `label`: a value that is either `0` for a negative review or `1` for a positive review.

## Preprocess

The next step is to load a DistilBERT tokenizer to preprocess the `text` field:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a preprocessing function to tokenize `text` and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def preprocess_function(examples):
...     return tokenizer(examples["text"], truncation=True)
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once:

```py
tokenized_imdb = imdb.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {0: "NEGATIVE", 1: "POSITIVE"}
>>> label2id = {"NEGATIVE": 0, "POSITIVE": 1}
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now!


key: _node_content
string_value:
{"id_": "7baf32fb-7075-400a-a83e-8905e874067d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4969f29-c2a9-4b84-a27c-b5506a8ac4f8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "14683a6ae77a3f02d3972fe6c3699fd4137fc6bb2d560c2541ff242a24be6adf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "119b7fb5-7644-4008-b468-71d222211db6", "node_type": "1", "metadata": {}, "hash": "49ca7d701c0816725ca87ef6d19f2889961420d973a7cf738b421786b87cdb82", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4526, "end_char_idx": 7682, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {0: "NEGATIVE", 1: "POSITIVE"}
>>> label2id = {"NEGATIVE": 0, "POSITIVE": 1}
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

>>> model = AutoModelForSequenceClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

<Tip>

[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it.


key: _node_content
string_value:
{"id_": "119b7fb5-7644-4008-b468-71d222211db6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7baf32fb-7075-400a-a83e-8905e874067d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "90d5fef50ae0d4b22113718cc0884419f127ed717555f883bab127060890de50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f1446ab1-9874-414b-8083-5d4367b8f7b9", "node_type": "1", "metadata": {}, "hash": "a02a13758076baf193f52c565ca4cd94d2e70031fe1570d637a18885cda6d7ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6347, "end_char_idx": 8106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

<Tip>

[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it. In this case, you don't need to specify a data collator explicitly.

</Tip>

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "f1446ab1-9874-414b-8083-5d4367b8f7b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "119b7fb5-7644-4008-b468-71d222211db6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4d48af34b1dd545e0f5e5f8f9d8d77557685e81ab3d81d9c8e7f05179d395f20", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9c07881e-7719-43f4-8e6d-5146c860b8c8", "node_type": "1", "metadata": {}, "hash": "285521b981712ea7e64628ca33239a27793b1e03088acdd4ef6cb59f92ade9bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6625, "end_char_idx": 9580, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

<Tip>

[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it. In this case, you don't need to specify a data collator explicitly.

</Tip>

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer
>>> import tensorflow as tf

>>> batch_size = 16
>>> num_epochs = 5
>>> batches_per_epoch = len(tokenized_imdb["train"]) // batch_size
>>> total_train_steps = int(batches_per_epoch * num_epochs)
>>> optimizer, schedule = create_optimizer(init_lr=2e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load DistilBERT with [`TFAutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_imdb["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_imdb["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method).


key: _node_content
string_value:
{"id_": "9c07881e-7719-43f4-8e6d-5146c860b8c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f1446ab1-9874-414b-8083-5d4367b8f7b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "77b7936873081c661c04b4de25d4674a773d14512fd5eff10fcee9bdc3c79c9c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8874f10e-6ad4-4363-a7ec-bee0b80059b0", "node_type": "1", "metadata": {}, "hash": "7a50748331295bcd0994e80d4a29de9a67d331131de0ae96bcc3673319e535f3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8108, "end_char_idx": 11133, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer
>>> import tensorflow as tf

>>> batch_size = 16
>>> num_epochs = 5
>>> batches_per_epoch = len(tokenized_imdb["train"]) // batch_size
>>> total_train_steps = int(batches_per_epoch * num_epochs)
>>> optimizer, schedule = create_optimizer(init_lr=2e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load DistilBERT with [`TFAutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_imdb["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_imdb["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!


key: _node_content
string_value:
{"id_": "8874f10e-6ad4-4363-a7ec-bee0b80059b0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9c07881e-7719-43f4-8e6d-5146c860b8c8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c5e1e233cdaa30a926d08b9a666ad1bab5574fdbc2b2845625963e7eb6cd815", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74c27d02-cfdf-4a91-b449-efbcf5583252", "node_type": "1", "metadata": {}, "hash": "0a2b117928ab9b5ba6db604303ff2a0a90432669fc21c51096ff8ba2f8f9ecf9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9581, "end_char_idx": 11945, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for text classification, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`].


key: _node_content
string_value:
{"id_": "74c27d02-cfdf-4a91-b449-efbcf5583252", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46d9b531-7674-446a-af93-d6cc3db1100f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4da200c08e9e1e3b9f34d00a1d1bc17e848c0874284c1df791de14d420b2b0c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8874f10e-6ad4-4363-a7ec-bee0b80059b0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification/", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "057bdea6d71be47eae1efbad2b092f090c554474a52dcc5eb277b95face20046", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10744, "end_char_idx": 13871, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for text classification, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for sentiment analysis with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model")
>>> classifier(text)
[{'label': 'POSITIVE', 'score': 0.9994940757751465}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("stevhliu/my_awesome_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_class_id = logits.argmax().item()
>>> model.config.id2label[predicted_class_id]
'POSITIVE'
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_model")
>>> inputs = tokenizer(text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained("stevhliu/my_awesome_model")
>>> logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
>>> model.config.id2label[predicted_class_id]
'POSITIVE'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "55ce67ed-cfb8-4c2d-9d8a-86628752ddfe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd4f1715-6b5e-44ff-8a77-c5b54c741943", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8d42b38b82e11b900cedc9efd7659866ddd376537389254ca45b36b4861975da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ee379ec-01b8-494f-8dda-67ca76654d04", "node_type": "1", "metadata": {}, "hash": "f4101d13ac743e5a7ba8b06700dbae65724d23051cafcc31edef369b6d97a6e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3265, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Knowledge Distillation for Computer Vision

[[open-in-colab]]

Knowledge distillation is a technique used to transfer knowledge from a larger, more complex model (teacher) to a smaller, simpler model (student). To distill knowledge from one model to another, we take a pre-trained teacher model trained on a certain task (image classification for this case) and randomly initialize a student model to be trained on image classification. Next, we train the student model to minimize the difference between it's outputs and the teacher's outputs, thus making it mimic the behavior. It was first introduced in [Distilling the Knowledge in a Neural Network by Hinton et al](https://arxiv.org/abs/1503.02531). In this guide, we will do task-specific knowledge distillation. We will use the [beans dataset](https://huggingface.co/datasets/beans) for this.

This guide demonstrates how you can distill a [fine-tuned ViT model](https://huggingface.co/merve/vit-mobilenet-beans-224) (teacher model) to a [MobileNet](https://huggingface.co/google/mobilenet_v2_1.4_224) (student model) using the [Trainer API](https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer) of 🤗 Transformers. 

Let's install the libraries needed for distillation and evaluating the process. 

```bash
pip install transformers datasets accelerate tensorboard evaluate --upgrade
```

In this example, we are using the `merve/beans-vit-224` model as teacher model. It's an image classification model, based on `google/vit-base-patch16-224-in21k` fine-tuned on beans dataset. We will distill this model to a randomly initialized MobileNetV2.

We will now load the dataset. 

```python
from datasets import load_dataset

dataset = load_dataset("beans")
```

We can use an image processor from either of the models, as in this case they return the same output with same resolution. We will use the `map()` method of `dataset` to apply the preprocessing to every split of the dataset. 

```python
from transformers import AutoImageProcessor
teacher_processor = AutoImageProcessor.from_pretrained("merve/beans-vit-224")

def process(examples):
    processed_inputs = teacher_processor(examples["image"])
    return processed_inputs

processed_datasets = dataset.map(process, batched=True)
```

Essentially, we want the student model (a randomly initialized MobileNet) to mimic the teacher model (fine-tuned vision transformer). To achieve this, we first get the logits output from the teacher and the student. Then, we divide each of them by the parameter `temperature` which controls the importance of each soft target. A parameter called `lambda` weighs the importance of the distillation loss. In this example, we will use `temperature=5` and `lambda=0.5`. We will use the Kullback-Leibler Divergence loss to compute the divergence between the student and teacher. Given two data P and Q, KL Divergence explains how much extra information we need to represent P using Q. If two are identical, their KL divergence is zero, as there's no other information needed to explain P from Q. Thus, in the context of knowledge distillation, KL divergence is useful.


```python
from transformers import TrainingArguments, Trainer
import torch
import torch.nn as nn
import torch.nn.functional as F


key: _node_content
string_value:
{"id_": "0ee379ec-01b8-494f-8dda-67ca76654d04", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd4f1715-6b5e-44ff-8a77-c5b54c741943", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8d42b38b82e11b900cedc9efd7659866ddd376537389254ca45b36b4861975da", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55ce67ed-cfb8-4c2d-9d8a-86628752ddfe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "28c26ba384ec9191cea7778d454939e7c8502477b92d074b047ca348e893b4a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4e45563-38af-4d96-9ec7-744e17567638", "node_type": "1", "metadata": {}, "hash": "ca5d245df04c246f84d8905fd1b793d20047ad56d69ec53d7cfd79e0310a923f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3137, "end_char_idx": 6506, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import TrainingArguments, Trainer
import torch
import torch.nn as nn
import torch.nn.functional as F


class ImageDistilTrainer(Trainer):
    def __init__(self, teacher_model=None, student_model=None, temperature=None, lambda_param=None,  *args, **kwargs):
        super().__init__(model=student_model, *args, **kwargs)
        self.teacher = teacher_model
        self.student = student_model
        self.loss_function = nn.KLDivLoss(reduction="batchmean")
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        self.teacher.to(device)
        self.teacher.eval()
        self.temperature = temperature
        self.lambda_param = lambda_param

    def compute_loss(self, student, inputs, return_outputs=False):
        student_output = self.student(**inputs)

        with torch.no_grad():
          teacher_output = self.teacher(**inputs)

        # Compute soft targets for teacher and student
        soft_teacher = F.softmax(teacher_output.logits / self.temperature, dim=-1)
        soft_student = F.log_softmax(student_output.logits / self.temperature, dim=-1)

        # Compute the loss
        distillation_loss = self.loss_function(soft_student, soft_teacher) * (self.temperature ** 2)

        # Compute the true label loss
        student_target_loss = student_output.loss

        # Calculate final loss
        loss = (1. - self.lambda_param) * student_target_loss + self.lambda_param * distillation_loss
        return (loss, student_output) if return_outputs else loss
```

We will now login to Hugging Face Hub so we can push our model to the Hugging Face Hub through the `Trainer`. 

```python
from huggingface_hub import notebook_login

notebook_login()
```

Let's set the `TrainingArguments`, the teacher model and the student model. 

```python
from transformers import AutoModelForImageClassification, MobileNetV2Config, MobileNetV2ForImageClassification

training_args = TrainingArguments(
    output_dir="my-awesome-model",
    num_train_epochs=30,
    fp16=True,
    logging_dir=f"{repo_name}/logs",
    logging_strategy="epoch",
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    report_to="tensorboard",
    push_to_hub=True,
    hub_strategy="every_save",
    hub_model_id=repo_name,
    )

num_labels = len(processed_datasets["train"].features["labels"].names)

# initialize models
teacher_model = AutoModelForImageClassification.from_pretrained(
    "merve/beans-vit-224",
    num_labels=num_labels,
    ignore_mismatched_sizes=True
)

# training MobileNetV2 from scratch
student_config = MobileNetV2Config()
student_config.num_labels = num_labels
student_model = MobileNetV2ForImageClassification(student_config)
```

We can use `compute_metrics` function to evaluate our model on the test set. This function will be used during the training process to compute the `accuracy` & `f1` of our model.

```python
import evaluate
import numpy as np

accuracy = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    acc = accuracy.compute(references=labels, predictions=np.argmax(predictions, axis=1))
    return {"accuracy": acc["accuracy"]}
```

Let's initialize the `Trainer` with the training arguments we defined. We will also initialize our data collator.


key: _node_content
string_value:
{"id_": "e4e45563-38af-4d96-9ec7-744e17567638", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd4f1715-6b5e-44ff-8a77-c5b54c741943", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8d42b38b82e11b900cedc9efd7659866ddd376537389254ca45b36b4861975da", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ee379ec-01b8-494f-8dda-67ca76654d04", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification/", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "17de528adaf8ec8fb171a9cce1bd5c6e1af3637b2273deea353a900087b542de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4800, "end_char_idx": 7801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from huggingface_hub import notebook_login

notebook_login()
```

Let's set the `TrainingArguments`, the teacher model and the student model. 

```python
from transformers import AutoModelForImageClassification, MobileNetV2Config, MobileNetV2ForImageClassification

training_args = TrainingArguments(
    output_dir="my-awesome-model",
    num_train_epochs=30,
    fp16=True,
    logging_dir=f"{repo_name}/logs",
    logging_strategy="epoch",
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    report_to="tensorboard",
    push_to_hub=True,
    hub_strategy="every_save",
    hub_model_id=repo_name,
    )

num_labels = len(processed_datasets["train"].features["labels"].names)

# initialize models
teacher_model = AutoModelForImageClassification.from_pretrained(
    "merve/beans-vit-224",
    num_labels=num_labels,
    ignore_mismatched_sizes=True
)

# training MobileNetV2 from scratch
student_config = MobileNetV2Config()
student_config.num_labels = num_labels
student_model = MobileNetV2ForImageClassification(student_config)
```

We can use `compute_metrics` function to evaluate our model on the test set. This function will be used during the training process to compute the `accuracy` & `f1` of our model.

```python
import evaluate
import numpy as np

accuracy = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    acc = accuracy.compute(references=labels, predictions=np.argmax(predictions, axis=1))
    return {"accuracy": acc["accuracy"]}
```

Let's initialize the `Trainer` with the training arguments we defined. We will also initialize our data collator.

```python
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator()
trainer = ImageDistilTrainer(
    student_model=student_model,
    teacher_model=teacher_model,
    training_args=training_args,
    train_dataset=processed_datasets["train"],
    eval_dataset=processed_datasets["validation"],
    data_collator=data_collator,
    tokenizer=teacher_processor,
    compute_metrics=compute_metrics,
    temperature=5,
    lambda_param=0.5
)
```

We can now train our model.

```python
trainer.train()
```

We can evaluate the model on the test set.

```python
trainer.evaluate(processed_datasets["test"])
```

On test set, our model reaches 72 percent accuracy. To have a sanity check over efficiency of distillation, we also trained MobileNet on the beans dataset from scratch with the same hyperparameters and observed 63 percent accuracy on the test set. We invite the readers to try different pre-trained teacher models, student architectures, distillation parameters and report their findings. The training logs and checkpoints for distilled model can be found in [this repository](https://huggingface.co/merve/vit-mobilenet-beans-224), and MobileNetV2 trained from scratch can be found in this [repository](https://huggingface.co/merve/resnet-mobilenet-beans-5).


key: _node_content
string_value:
{"id_": "391d1412-d46e-49d4-bc12-e5a6983abfeb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e95428b-b85a-4121-b33d-2c8d9a691bef", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9deb2ab45b064aa33d95ced9520e1a4d8616bc857f1e0ed6c9909a88fc604670", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ccda4c3-bc22-4f7f-ac1a-ddf05147851a", "node_type": "1", "metadata": {}, "hash": "1767be3d4528c63af3a20d0e2272351bb3b493bf92d4441a2cfd515980d2c582", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Zero-shot image classification

[[open-in-colab]]

Zero-shot image classification is a task that involves classifying images into different categories using a model that was
not explicitly trained on data containing labeled examples from those specific categories.

Traditionally, image classification requires training a model on a specific set of labeled images, and this model learns to
"map" certain image features to labels. When there's a need to use such model for a classification task that introduces a
new set of labels, fine-tuning is required to "recalibrate" the model.

In contrast, zero-shot or open vocabulary image classification models are typically multi-modal models that have been trained on a large
dataset of images and associated descriptions. These models learn aligned vision-language representations that can be used for many downstream tasks including zero-shot image classification.

This is a more flexible approach to image classification that allows models to generalize to new and unseen categories
without the need for additional training data and enables users to query images with free-form text descriptions of their target objects .

In this guide you'll learn how to:

* create a zero-shot image classification pipeline
* run zero-shot image classification inference by hand

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q "transformers[torch]" pillow
```

## Zero-shot image classification pipeline

The simplest way to try out inference with a model supporting zero-shot image classification is to use the corresponding [`pipeline`].
Instantiate a pipeline from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=zero-shot-image-classification&sort=downloads):

```python
>>> from transformers import pipeline

>>> checkpoint = "openai/clip-vit-large-patch14"
>>> detector = pipeline(model=checkpoint, task="zero-shot-image-classification")
```

Next, choose an image you'd like to classify.

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/g8oS8-82DxI/download?ixid=MnwxMjA3fDB8MXx0b3BpY3x8SnBnNktpZGwtSGt8fHx8fDJ8fDE2NzgxMDYwODc&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/owl.jpg" alt="Photo of an owl"/>
</div>

Pass the image and the candidate object labels to the pipeline. Here we pass the image directly; other suitable options
include a local path to an image or an image url.
The candidate labels can be simple words like in this example, or more descriptive.

```py
>>> predictions = detector(image, candidate_labels=["fox", "bear", "seagull", "owl"])
>>> predictions
[{'score': 0.9996670484542847, 'label': 'owl'},
 {'score': 0.000199399160919711, 'label': 'seagull'},
 {'score': 7.392891711788252e-05, 'label': 'fox'},
 {'score': 5.96074532950297e-05, 'label': 'bear'}]
```

## Zero-shot image classification by hand

Now that you've seen how to use the zero-shot image classification pipeline, let's take a look how you can run zero-shot
image classification manually.


key: _node_content
string_value:
{"id_": "9ccda4c3-bc22-4f7f-ac1a-ddf05147851a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e95428b-b85a-4121-b33d-2c8d9a691bef", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9deb2ab45b064aa33d95ced9520e1a4d8616bc857f1e0ed6c9909a88fc604670", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "391d1412-d46e-49d4-bc12-e5a6983abfeb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "399ca5c1d46de2451928422af724d067beaf996f41c457bf0e2c2fb65e4d1e7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a328411-79d1-4f17-85d6-7e32d9f33999", "node_type": "1", "metadata": {}, "hash": "50b3783ae1d12f9b73763d3c05c29bf00ed838cff6ec905a114117423236e6b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2020, "end_char_idx": 4604, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/g8oS8-82DxI/download?ixid=MnwxMjA3fDB8MXx0b3BpY3x8SnBnNktpZGwtSGt8fHx8fDJ8fDE2NzgxMDYwODc&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/owl.jpg" alt="Photo of an owl"/>
</div>

Pass the image and the candidate object labels to the pipeline. Here we pass the image directly; other suitable options
include a local path to an image or an image url.
The candidate labels can be simple words like in this example, or more descriptive.

```py
>>> predictions = detector(image, candidate_labels=["fox", "bear", "seagull", "owl"])
>>> predictions
[{'score': 0.9996670484542847, 'label': 'owl'},
 {'score': 0.000199399160919711, 'label': 'seagull'},
 {'score': 7.392891711788252e-05, 'label': 'fox'},
 {'score': 5.96074532950297e-05, 'label': 'bear'}]
```

## Zero-shot image classification by hand

Now that you've seen how to use the zero-shot image classification pipeline, let's take a look how you can run zero-shot
image classification manually.

Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=zero-shot-image-classification&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoProcessor, AutoModelForZeroShotImageClassification

>>> model = AutoModelForZeroShotImageClassification.from_pretrained(checkpoint)
>>> processor = AutoProcessor.from_pretrained(checkpoint)
```

Let's take a different image to switch things up.

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/xBRQfR2bqNI/download?ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjc4Mzg4ODEx&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg" alt="Photo of a car"/>
</div>

Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the
image for the model by resizing and normalizing it, and a tokenizer that takes care of the text inputs.

```py
>>> candidate_labels = ["tree", "car", "bike", "cat"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.'


key: _node_content
string_value:
{"id_": "7a328411-79d1-4f17-85d6-7e32d9f33999", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e95428b-b85a-4121-b33d-2c8d9a691bef", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9deb2ab45b064aa33d95ced9520e1a4d8616bc857f1e0ed6c9909a88fc604670", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ccda4c3-bc22-4f7f-ac1a-ddf05147851a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification/", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52809fd3b00dbcdda76e15ddfeb64bf9efa264129ada1e63b0245b3bef9ca017", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3250, "end_char_idx": 5350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=zero-shot-image-classification&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoProcessor, AutoModelForZeroShotImageClassification

>>> model = AutoModelForZeroShotImageClassification.from_pretrained(checkpoint)
>>> processor = AutoProcessor.from_pretrained(checkpoint)
```

Let's take a different image to switch things up.

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/xBRQfR2bqNI/download?ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjc4Mzg4ODEx&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg" alt="Photo of a car"/>
</div>

Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the
image for the model by resizing and normalizing it, and a tokenizer that takes care of the text inputs.

```py
>>> candidate_labels = ["tree", "car", "bike", "cat"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]
>>> inputs = processor(images=image, text=candidate_labels, return_tensors="pt", padding=True)
```

Pass the inputs through the model, and post-process the results:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits = outputs.logits_per_image[0]
>>> probs = logits.softmax(dim=-1).numpy()
>>> scores = probs.tolist()

>>> result = [
...     {"score": score, "label": candidate_label}
...     for score, candidate_label in sorted(zip(probs, candidate_labels), key=lambda x: -x[0])
... ]

>>> result
[{'score': 0.998572, 'label': 'car'},
 {'score': 0.0010570387, 'label': 'bike'},
 {'score': 0.0003393686, 'label': 'tree'},
 {'score': 3.1572064e-05, 'label': 'cat'}]
```


key: _node_content
string_value:
{"id_": "f84c6eae-e6af-4303-b8d8-e7b382f6b9c2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3a9c22f1-2837-49b4-bd58-5756ee1350af", "node_type": "1", "metadata": {}, "hash": "c57eb3e8f8f601c7c77ae445c2f7e1a83ae1b629cfa57d4d8079823b0f06dd8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Visual Question Answering

[[open-in-colab]]

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image. 
The input to models supporting this task is typically a combination of an image and a question, and the output is an 
answer expressed in natural language.

Some noteworthy use case examples for VQA include:
* Accessibility applications for visually impaired individuals.
* Education: posing questions about visual materials presented in lectures or textbooks. VQA can also be utilized in interactive museum exhibits or historical sites.
* Customer service and e-commerce: VQA can enhance user experience by letting users ask questions about products. 
* Image retrieval: VQA models can be used to retrieve images with specific characteristics. For example, the user can ask "Is there a dog?" to find all images with dogs from a set of images.

In this guide you'll learn how to:

- Fine-tune a classification VQA model, specifically [ViLT](../model_doc/vilt), on the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa).
- Use your fine-tuned ViLT for inference.
- Run zero-shot VQA inference with a generative model, like BLIP-2.

## Fine-tuning ViLT

ViLT model incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design for 
Vision-and-Language Pre-training (VLP). This model can be used for several downstream tasks. For the VQA task, a classifier 
head is placed on top (a linear layer on top of the final hidden state of the `[CLS]` token) and randomly initialized. 
Visual Question Answering is thus treated as a **classification problem**.

More recent models, such as BLIP, BLIP-2, and InstructBLIP, treat VQA as a generative task. Later in this guide we 
illustrate how to use them for zero-shot VQA inference. 

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q transformers datasets
```

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define the model checkpoint as a global variable.

```py
>>> model_checkpoint = "dandelin/vilt-b32-mlm"
```

## Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering `Graphcore/vqa` dataset. 
You can find the full dataset on [🤗 Hub](https://huggingface.co/datasets/Graphcore/vqa).

As an alternative to the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa), you can download the 
same data manually from the official [VQA dataset page](https://visualqa.org/download.html). If you prefer to follow the 
tutorial with your custom data, check out how to [Create an image dataset](https://huggingface.co/docs/datasets/image_dataset#loading-script)
guide in the 🤗 Datasets documentation.


key: _node_content
string_value:
{"id_": "3a9c22f1-2837-49b4-bd58-5756ee1350af", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f84c6eae-e6af-4303-b8d8-e7b382f6b9c2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9838bc1503d39b5d52a8310d7eeea348f72f5a89ff4c71ce37d24bd3990677b1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c29f18fd-1c89-40ae-bea2-35a38fb42876", "node_type": "1", "metadata": {}, "hash": "0acb8adae8bd87e2910c4ca9df30d516a69aad0f0ce8d91dd6dad342761e8321", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1375, "end_char_idx": 3477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model can be used for several downstream tasks. For the VQA task, a classifier 
head is placed on top (a linear layer on top of the final hidden state of the `[CLS]` token) and randomly initialized. 
Visual Question Answering is thus treated as a **classification problem**.

More recent models, such as BLIP, BLIP-2, and InstructBLIP, treat VQA as a generative task. Later in this guide we 
illustrate how to use them for zero-shot VQA inference. 

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q transformers datasets
```

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define the model checkpoint as a global variable.

```py
>>> model_checkpoint = "dandelin/vilt-b32-mlm"
```

## Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering `Graphcore/vqa` dataset. 
You can find the full dataset on [🤗 Hub](https://huggingface.co/datasets/Graphcore/vqa).

As an alternative to the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa), you can download the 
same data manually from the official [VQA dataset page](https://visualqa.org/download.html). If you prefer to follow the 
tutorial with your custom data, check out how to [Create an image dataset](https://huggingface.co/docs/datasets/image_dataset#loading-script)
guide in the 🤗 Datasets documentation.  

Let's load the first 200 examples from the validation split and explore the dataset's features:  

```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("Graphcore/vqa", split="validation[:200]")
>>> dataset
Dataset({
    features: ['question', 'question_type', 'question_id', 'image_id', 'answer_type', 'label'],
    num_rows: 200
})
```

Let's take a look at an example to understand the dataset's features:

```py
>>> dataset[0]
{'question': 'Where is he looking?


key: _node_content
string_value:
{"id_": "c29f18fd-1c89-40ae-bea2-35a38fb42876", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3a9c22f1-2837-49b4-bd58-5756ee1350af", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3faabd46885434082b8a66ef5e06047dbd8e8424ce35375189c84211a01d2ac5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a774d27c-27c1-43e6-bdf7-7063729eaee6", "node_type": "1", "metadata": {}, "hash": "a60eaf7042a93c1f1b08de60c13fafbe9cdcf01194db13ac0cc8e849cc5f4068", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1830, "end_char_idx": 4618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q transformers datasets
```

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define the model checkpoint as a global variable.

```py
>>> model_checkpoint = "dandelin/vilt-b32-mlm"
```

## Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering `Graphcore/vqa` dataset. 
You can find the full dataset on [🤗 Hub](https://huggingface.co/datasets/Graphcore/vqa).

As an alternative to the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa), you can download the 
same data manually from the official [VQA dataset page](https://visualqa.org/download.html). If you prefer to follow the 
tutorial with your custom data, check out how to [Create an image dataset](https://huggingface.co/docs/datasets/image_dataset#loading-script)
guide in the 🤗 Datasets documentation.  

Let's load the first 200 examples from the validation split and explore the dataset's features:  

```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("Graphcore/vqa", split="validation[:200]")
>>> dataset
Dataset({
    features: ['question', 'question_type', 'question_id', 'image_id', 'answer_type', 'label'],
    num_rows: 200
})
```

Let's take a look at an example to understand the dataset's features:

```py
>>> dataset[0]
{'question': 'Where is he looking?',
 'question_type': 'none of the above',
 'question_id': 262148000,
 'image_id': '/root/.cache/huggingface/datasets/downloads/extracted/ca733e0e000fb2d7a09fbcc94dbfe7b5a30750681d0e965f8e0a23b1c2f98c75/val2014/COCO_val2014_000000262148.jpg',
 'answer_type': 'other',
 'label': {'ids': ['at table', 'down', 'skateboard', 'table'],
  'weights': [0.30000001192092896,
   1.0,
   0.30000001192092896,
   0.30000001192092896]}}
```

The features relevant to the task include: 
* `question`: the question to be answered from the image
* `image_id`: the path to the image the question refers to
* `label`: the annotations

We can remove the rest of the features as they won't be necessary: 

```py 
>>> dataset = dataset.remove_columns(['question_type', 'question_id', 'answer_type'])
```

As you can see, the `label` feature contains several answers to the same question (called `ids` here) collected by different human annotators. 
This is because the answer to a question can be subjective. In this case, the question is "where is he looking?". Some people 
annotated this with "down", others with "at table", another one with "skateboard", etc.


key: _node_content
string_value:
{"id_": "a774d27c-27c1-43e6-bdf7-7063729eaee6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c29f18fd-1c89-40ae-bea2-35a38fb42876", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d7797d28d9ab0c69d7ce80ade083b14bc480c6bbf6bcd859bbee131d73f7a97e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b138b231-66fb-4fbd-a3f6-f9186bd82222", "node_type": "1", "metadata": {}, "hash": "847bf8da481c4cf10b0f7305232170266be8f2595332e18eab5160d0faaf62e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3477, "end_char_idx": 6306, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
',
 'question_type': 'none of the above',
 'question_id': 262148000,
 'image_id': '/root/.cache/huggingface/datasets/downloads/extracted/ca733e0e000fb2d7a09fbcc94dbfe7b5a30750681d0e965f8e0a23b1c2f98c75/val2014/COCO_val2014_000000262148.jpg',
 'answer_type': 'other',
 'label': {'ids': ['at table', 'down', 'skateboard', 'table'],
  'weights': [0.30000001192092896,
   1.0,
   0.30000001192092896,
   0.30000001192092896]}}
```

The features relevant to the task include: 
* `question`: the question to be answered from the image
* `image_id`: the path to the image the question refers to
* `label`: the annotations

We can remove the rest of the features as they won't be necessary: 

```py 
>>> dataset = dataset.remove_columns(['question_type', 'question_id', 'answer_type'])
```

As you can see, the `label` feature contains several answers to the same question (called `ids` here) collected by different human annotators. 
This is because the answer to a question can be subjective. In this case, the question is "where is he looking?". Some people 
annotated this with "down", others with "at table", another one with "skateboard", etc. 

Take a look at the image and consider which answer would you give:

```python
>>> from PIL import Image

>>> image = Image.open(dataset[0]['image_id'])
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/vqa-example.png" alt="VQA Image Example"/>
</div>

Due to the questions' and answers' ambiguity, datasets like this are treated as a multi-label classification problem (as 
multiple answers are possibly valid). Moreover, rather than just creating a one-hot encoded vector, one creates a 
soft encoding, based on the number of times a certain answer appeared in the annotations.

For instance, in the example above, because the answer "down" is selected way more often than other answers, it has a 
score (called `weight` in the dataset) of 1.0, and the rest of the answers have scores < 1.0. 

To later instantiate the model with an appropriate classification head, let's create two dictionaries: one that maps 
the label name to an integer and vice versa:

```py
>>> import itertools

>>> labels = [item['ids'] for item in dataset['label']]
>>> flattened_labels = list(itertools.chain(*labels))
>>> unique_labels = list(set(flattened_labels))

>>> label2id = {label: idx for idx, label in enumerate(unique_labels)}
>>> id2label = {idx: label for label, idx in label2id.items()} 
```

Now that we have the mappings, we can replace the string answers with their ids, and flatten the dataset for a more convenient further preprocessing. 

```python
>>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs


key: _node_content
string_value:
{"id_": "b138b231-66fb-4fbd-a3f6-f9186bd82222", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a774d27c-27c1-43e6-bdf7-7063729eaee6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4fc41ef2d34e9e4d4b7479359cb45b0a0bdbb158b782608b77611a37bf48927c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a62e19cf-cabe-40ed-9fb4-65adf93fd5f9", "node_type": "1", "metadata": {}, "hash": "becfcde5f4e9112fe428353d9a77713fe845b64fe9f9002de014d9b90f9ba48b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4621, "end_char_idx": 7755, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Take a look at the image and consider which answer would you give:

```python
>>> from PIL import Image

>>> image = Image.open(dataset[0]['image_id'])
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/vqa-example.png" alt="VQA Image Example"/>
</div>

Due to the questions' and answers' ambiguity, datasets like this are treated as a multi-label classification problem (as 
multiple answers are possibly valid). Moreover, rather than just creating a one-hot encoded vector, one creates a 
soft encoding, based on the number of times a certain answer appeared in the annotations.

For instance, in the example above, because the answer "down" is selected way more often than other answers, it has a 
score (called `weight` in the dataset) of 1.0, and the rest of the answers have scores < 1.0. 

To later instantiate the model with an appropriate classification head, let's create two dictionaries: one that maps 
the label name to an integer and vice versa:

```py
>>> import itertools

>>> labels = [item['ids'] for item in dataset['label']]
>>> flattened_labels = list(itertools.chain(*labels))
>>> unique_labels = list(set(flattened_labels))

>>> label2id = {label: idx for idx, label in enumerate(unique_labels)}
>>> id2label = {idx: label for label, idx in label2id.items()} 
```

Now that we have the mappings, we can replace the string answers with their ids, and flatten the dataset for a more convenient further preprocessing. 

```python
>>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs


>>> dataset = dataset.map(replace_ids)
>>> flat_dataset = dataset.flatten()
>>> flat_dataset.features
{'question': Value(dtype='string', id=None),
 'image_id': Value(dtype='string', id=None),
 'label.ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),
 'label.weights': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None)}
```

## Preprocessing data

The next step is to load a ViLT processor to prepare the image and text data for the model. 
[`ViltProcessor`] wraps a BERT tokenizer and ViLT image processor into a convenient single processor:

```py 
>>> from transformers import ViltProcessor

>>> processor = ViltProcessor.from_pretrained(model_checkpoint)
```

To preprocess the data we need to encode the images and questions using the [`ViltProcessor`]. The processor will use 
the [`BertTokenizerFast`] to tokenize the text and create `input_ids`, `attention_mask` and `token_type_ids` for the text data. 
As for images, the processor will leverage [`ViltImageProcessor`] to resize and normalize the image, and create `pixel_values` and `pixel_mask`.

All these preprocessing steps are done under the hood, we only need to call the `processor`. However, we still need to 
prepare the target labels. In this representation, each element corresponds to a possible answer (label). For correct answers, the element holds 
their respective score (weight), while the remaining elements are set to zero.


key: _node_content
string_value:
{"id_": "a62e19cf-cabe-40ed-9fb4-65adf93fd5f9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b138b231-66fb-4fbd-a3f6-f9186bd82222", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b629818e5a92d2b6d7c9f26f0e479edaa4a3dd2e60b61e56c2c32012750ec710", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "630cd81f-52fa-4820-8702-d51d48e3c7fa", "node_type": "1", "metadata": {}, "hash": "f8255841ffdfce85f3c0721f1214c436bf83f0228fa8a8c305268984bab2b930", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6171, "end_char_idx": 9516, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs


>>> dataset = dataset.map(replace_ids)
>>> flat_dataset = dataset.flatten()
>>> flat_dataset.features
{'question': Value(dtype='string', id=None),
 'image_id': Value(dtype='string', id=None),
 'label.ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),
 'label.weights': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None)}
```

## Preprocessing data

The next step is to load a ViLT processor to prepare the image and text data for the model. 
[`ViltProcessor`] wraps a BERT tokenizer and ViLT image processor into a convenient single processor:

```py 
>>> from transformers import ViltProcessor

>>> processor = ViltProcessor.from_pretrained(model_checkpoint)
```

To preprocess the data we need to encode the images and questions using the [`ViltProcessor`]. The processor will use 
the [`BertTokenizerFast`] to tokenize the text and create `input_ids`, `attention_mask` and `token_type_ids` for the text data. 
As for images, the processor will leverage [`ViltImageProcessor`] to resize and normalize the image, and create `pixel_values` and `pixel_mask`.

All these preprocessing steps are done under the hood, we only need to call the `processor`. However, we still need to 
prepare the target labels. In this representation, each element corresponds to a possible answer (label). For correct answers, the element holds 
their respective score (weight), while the remaining elements are set to zero.

The following function applies the `processor` to the images and questions and formats the labels as described above:

```py
>>> import torch

>>> def preprocess_data(examples):
...     image_paths = examples['image_id']
...     images = [Image.open(image_path) for image_path in image_paths]
...     texts = examples['question']    

...     encoding = processor(images, texts, padding="max_length", truncation=True, return_tensors="pt")

...     for k, v in encoding.items():
...           encoding[k] = v.squeeze()
    
...     targets = []

...     for labels, scores in zip(examples['label.ids'], examples['label.weights']):
...         target = torch.zeros(len(id2label))

...         for label, score in zip(labels, scores):
...             target[label] = score
      
...         targets.append(target)

...     encoding["labels"] = targets
    
...     return encoding
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.map`] function. You can speed up `map` by 
setting `batched=True` to process multiple elements of the dataset at once. At this point, feel free to remove the columns you don't need.

```py
>>> processed_dataset = flat_dataset.map(preprocess_data, batched=True, remove_columns=['question','question_type',  'question_id', 'image_id', 'answer_type', 'label.ids', 'label.weights'])
>>> processed_dataset
Dataset({
    features: ['input_ids', 'token_type_ids', 'attention_mask', 'pixel_values', 'pixel_mask', 'labels'],
    num_rows: 200
})
```

As a final step, create a batch of examples using [`DefaultDataCollator`]:

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

## Train the model

You’re ready to start training your model now!


key: _node_content
string_value:
{"id_": "630cd81f-52fa-4820-8702-d51d48e3c7fa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a62e19cf-cabe-40ed-9fb4-65adf93fd5f9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8fec4526323723001f91768f1f0196ed933bfadf1d2b012a661b5b517e03b899", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d24b4664-6698-4263-b065-913109c26a09", "node_type": "1", "metadata": {}, "hash": "d834cd5660a1e4fb5399a7713ea8000b62145920c168ee083a0488363d97b325", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8745, "end_char_idx": 11830, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can speed up `map` by 
setting `batched=True` to process multiple elements of the dataset at once. At this point, feel free to remove the columns you don't need.

```py
>>> processed_dataset = flat_dataset.map(preprocess_data, batched=True, remove_columns=['question','question_type',  'question_id', 'image_id', 'answer_type', 'label.ids', 'label.weights'])
>>> processed_dataset
Dataset({
    features: ['input_ids', 'token_type_ids', 'attention_mask', 'pixel_values', 'pixel_mask', 'labels'],
    num_rows: 200
})
```

As a final step, create a batch of examples using [`DefaultDataCollator`]:

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

## Train the model

You’re ready to start training your model now! Load ViLT with [`ViltForQuestionAnswering`]. Specify the number of labels 
along with the label mappings:

```py
>>> from transformers import ViltForQuestionAnswering

>>> model = ViltForQuestionAnswering.from_pretrained(model_checkpoint, num_labels=len(id2label), id2label=id2label, label2id=label2id)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]:

```py
>>> from transformers import TrainingArguments

>>> repo_id = "MariaK/vilt_finetuned_200"

>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...     logging_steps=50,
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
```

2. Pass the training arguments to [`Trainer`] along with the model, dataset, processor, and data collator.

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=processed_dataset,
...     tokenizer=processor,
... )
```

3. Call [`~Trainer.train`] to finetune your model.

```py
>>> trainer.train() 
```

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method to share your final model on the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

Now that you have fine-tuned a ViLT model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your fine-tuned model for inference is to use it in a [`Pipeline`].

```py
>>> from transformers import pipeline

>>> pipe = pipeline("visual-question-answering", model="MariaK/vilt_finetuned_200")
```

The model in this guide has only been trained on 200 examples, so don't expect a lot from it. Let's see if it at least 
learned something from the data and take the first example from the dataset to illustrate inference:

```py
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
>>> print(question)
>>> pipe(image, question, top_k=1)
"Where is he looking?"
[{'score': 0.5498199462890625, 'answer': 'down'}]
```

Even though not very confident, the model indeed has learned something.


key: _node_content
string_value:
{"id_": "d24b4664-6698-4263-b065-913109c26a09", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "630cd81f-52fa-4820-8702-d51d48e3c7fa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1a4121350f84a09c98ba9a5e86d82dc1f182bccd243657f6b4302a8ce64cf779", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "edc0fd09-2ae8-49d5-ae44-0086b2bc4dde", "node_type": "1", "metadata": {}, "hash": "e674133eefc00b096425314c7c1f55afe77986fa70e3d17259aa0990bd9b8935", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10348, "end_char_idx": 13430, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass the training arguments to [`Trainer`] along with the model, dataset, processor, and data collator.

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=processed_dataset,
...     tokenizer=processor,
... )
```

3. Call [`~Trainer.train`] to finetune your model.

```py
>>> trainer.train() 
```

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method to share your final model on the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

Now that you have fine-tuned a ViLT model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your fine-tuned model for inference is to use it in a [`Pipeline`].

```py
>>> from transformers import pipeline

>>> pipe = pipeline("visual-question-answering", model="MariaK/vilt_finetuned_200")
```

The model in this guide has only been trained on 200 examples, so don't expect a lot from it. Let's see if it at least 
learned something from the data and take the first example from the dataset to illustrate inference:

```py
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
>>> print(question)
>>> pipe(image, question, top_k=1)
"Where is he looking?"
[{'score': 0.5498199462890625, 'answer': 'down'}]
```

Even though not very confident, the model indeed has learned something. With more examples and longer training, you'll get far better results!

You can also manually replicate the results of the pipeline if you'd like:
1. Take an image and a question, prepare them for the model using the processor from your model.
2. Forward the result or preprocessing through the model.
3. From the logits, get the most likely answer's id, and find the actual answer in the `id2label`.

```py
>>> processor = ViltProcessor.from_pretrained("MariaK/vilt_finetuned_200")

>>> image = Image.open(example['image_id'])
>>> question = example['question']

>>> # prepare inputs
>>> inputs = processor(image, question, return_tensors="pt")

>>> model = ViltForQuestionAnswering.from_pretrained("MariaK/vilt_finetuned_200")

>>> # forward pass
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits = outputs.logits
>>> idx = logits.argmax(-1).item()
>>> print("Predicted answer:", model.config.id2label[idx])
Predicted answer: down
```

## Zero-shot VQA

The previous model treated VQA as a classification task. Some recent models, such as BLIP, BLIP-2, and InstructBLIP approach 
VQA as a generative task. Let's take [BLIP-2](../model_doc/blip-2) as an example. It introduced a new visual-language pre-training 
paradigm in which any combination of pre-trained vision encoder and LLM can be used (learn more in the [BLIP-2 blog post](https://huggingface.co/blog/blip-2)). 
This enables achieving state-of-the-art results on multiple visual-language tasks including visual question answering. 

Let's illustrate how you can use this model for VQA. First, let's load the model.


key: _node_content
string_value:
{"id_": "edc0fd09-2ae8-49d5-ae44-0086b2bc4dde", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d24b4664-6698-4263-b065-913109c26a09", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4699ba06daa6356274c146bf25df1d05e8e760619477c0b57c81297ce8383bf8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee57f3bd-c602-4a95-bc83-1c8ae94b5eaa", "node_type": "1", "metadata": {}, "hash": "be8b95418d0a422bce27dba3cbf966c13cc716a308af99ffa88e44b601d5f4d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11831, "end_char_idx": 15013, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With more examples and longer training, you'll get far better results!

You can also manually replicate the results of the pipeline if you'd like:
1. Take an image and a question, prepare them for the model using the processor from your model.
2. Forward the result or preprocessing through the model.
3. From the logits, get the most likely answer's id, and find the actual answer in the `id2label`.

```py
>>> processor = ViltProcessor.from_pretrained("MariaK/vilt_finetuned_200")

>>> image = Image.open(example['image_id'])
>>> question = example['question']

>>> # prepare inputs
>>> inputs = processor(image, question, return_tensors="pt")

>>> model = ViltForQuestionAnswering.from_pretrained("MariaK/vilt_finetuned_200")

>>> # forward pass
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits = outputs.logits
>>> idx = logits.argmax(-1).item()
>>> print("Predicted answer:", model.config.id2label[idx])
Predicted answer: down
```

## Zero-shot VQA

The previous model treated VQA as a classification task. Some recent models, such as BLIP, BLIP-2, and InstructBLIP approach 
VQA as a generative task. Let's take [BLIP-2](../model_doc/blip-2) as an example. It introduced a new visual-language pre-training 
paradigm in which any combination of pre-trained vision encoder and LLM can be used (learn more in the [BLIP-2 blog post](https://huggingface.co/blog/blip-2)). 
This enables achieving state-of-the-art results on multiple visual-language tasks including visual question answering. 

Let's illustrate how you can use this model for VQA. First, let's load the model. Here we'll explicitly send the model to a 
GPU, if available, which we didn't need to do earlier when training, as [`Trainer`] handles this automatically: 

```py
>>> from transformers import AutoProcessor, Blip2ForConditionalGeneration
>>> import torch

>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
```

The model takes image and text as input, so let's use the exact same image/question pair from the first example in the VQA dataset: 

```py 
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
```

To use BLIP-2 for visual question answering task, the textual prompt has to follow a specific format: `Question: {} Answer:`.

```py
>>> prompt = f"Question: {question} Answer:" 
```

Now we need to preprocess the image/prompt with the model's processor, pass the processed input through the model, and decode the output:

```py
>>> inputs = processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)

>>> generated_ids = model.generate(**inputs, max_new_tokens=10)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
"He is looking at the crowd" 
```

As you can see, the model recognized the crowd, and the direction of the face (looking down), however, it seems to miss 
the fact the crowd is behind the skater.


key: _node_content
string_value:
{"id_": "ee57f3bd-c602-4a95-bc83-1c8ae94b5eaa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77f501f3-2367-464d-920f-20999725ca2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d90dd6552ac0481b36fe789291de3f4ec17e1828bd5fc1dac06bf48d87165306", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "edc0fd09-2ae8-49d5-ae44-0086b2bc4dde", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering/", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "122b593541c0ef38e6c809a289c0e0c07b267a38b03763ceed377440bff2bb17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13402, "end_char_idx": 15138, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
First, let's load the model. Here we'll explicitly send the model to a 
GPU, if available, which we didn't need to do earlier when training, as [`Trainer`] handles this automatically: 

```py
>>> from transformers import AutoProcessor, Blip2ForConditionalGeneration
>>> import torch

>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
```

The model takes image and text as input, so let's use the exact same image/question pair from the first example in the VQA dataset: 

```py 
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
```

To use BLIP-2 for visual question answering task, the textual prompt has to follow a specific format: `Question: {} Answer:`.

```py
>>> prompt = f"Question: {question} Answer:" 
```

Now we need to preprocess the image/prompt with the model's processor, pass the processed input through the model, and decode the output:

```py
>>> inputs = processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)

>>> generated_ids = model.generate(**inputs, max_new_tokens=10)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
"He is looking at the crowd" 
```

As you can see, the model recognized the crowd, and the direction of the face (looking down), however, it seems to miss 
the fact the crowd is behind the skater. Still, in cases where acquiring human-annotated datasets is not feasible, this 
approach can quickly produce useful results.


key: _node_content
string_value:
{"id_": "11235822-c589-4102-bf79-91eb89682524", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "accb8ce4-acc8-452a-9206-4cbe936d0eab", "node_type": "1", "metadata": {}, "hash": "f07cf64d6a3f473b4ed9581a79a687b78d7edcf3fe101b81cfc7d3baac827e1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3039, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image classification

[[open-in-colab]]

<Youtube id="tjAIM7BOYhw"/>

Image classification assigns a label or class to an image. Unlike text or audio classification, the inputs are the
pixel values that comprise an image. There are many applications for image classification, such as detecting damage
after a natural disaster, monitoring crop health, or helping screen medical images for signs of disease.

This guide illustrates how to:

1. Fine-tune [ViT](model_doc/vit) on the [Food-101](https://huggingface.co/datasets/food101) dataset to classify a food item in an image.
2. Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-classification)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate accelerate pillow torchvision scikit-learn
```

We encourage you to log in to your Hugging Face account to upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load Food-101 dataset

Start by loading a smaller subset of the Food-101 dataset from the 🤗 Datasets library. This will give you a chance to
experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> food = load_dataset("food101", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> food = food.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> food["train"][0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x512 at 0x7F52AFC8AC50>,
 'label': 79}
```

Each example in the dataset has two fields:

- `image`: a PIL image of the food item
- `label`: the label class of the food item

To make it easier for the model to get the label name from the label id, create a dictionary that maps the label name
to an integer and vice versa:

```py
>>> labels = food["train"].features["label"].names
>>> label2id, id2label = dict(), dict()
>>> for i, label in enumerate(labels):
...     label2id[label] = str(i)
...     id2label[str(i)] = label
```

Now you can convert the label id to a label name:

```py
>>> id2label[str(79)]
'prime_rib'
```

## Preprocess

The next step is to load a ViT image processor to process the image into a tensor:

```py
>>> from transformers import AutoImageProcessor

>>> checkpoint = "google/vit-base-patch16-224-in21k"
>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
```

<frameworkcontent>
<pt>
Apply some image transformations to the images to make the model more robust against overfitting. Here you'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module, but you can also use any image library you like.


key: _node_content
string_value:
{"id_": "accb8ce4-acc8-452a-9206-4cbe936d0eab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "11235822-c589-4102-bf79-91eb89682524", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4619397f1e5ae396e8b72c1290f83eb7df0a314e01adad42e20334fd43437564", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5462b7e-dfee-464c-a832-05c895076ece", "node_type": "1", "metadata": {}, "hash": "f08f3d0f622ac08c0a2924aab247c9ae7ab6c449789efc2936b1da623800c471", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2887, "end_char_idx": 6194, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here you'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module, but you can also use any image library you like.

Crop a random part of the image, resize it, and normalize it with the image mean and standard deviation:

```py
>>> from torchvision.transforms import RandomResizedCrop, Compose, Normalize, ToTensor

>>> normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )
>>> _transforms = Compose([RandomResizedCrop(size), ToTensor(), normalize])
```

Then create a preprocessing function to apply the transforms and return the `pixel_values` - the inputs to the model - of the image:

```py
>>> def transforms(examples):
...     examples["pixel_values"] = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     del examples["image"]
...     return examples
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.with_transform`] method. The transforms are applied on the fly when you load an element of the dataset:

```py
>>> food = food.with_transform(transforms)
```

Now create a batch of examples using [`DefaultDataCollator`]. Unlike other data collators in 🤗 Transformers, the `DefaultDataCollator` does not apply additional preprocessing such as padding.

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```
</pt>
</frameworkcontent>


<frameworkcontent>
<tf>

To avoid overfitting and to make the model more robust, add some data augmentation to the training part of the dataset.
Here we use Keras preprocessing layers to define the transformations for the training data (includes data augmentation),
and transformations for the validation data (only center cropping, resizing and normalizing). You can use `tf.image`or
any other library you prefer.

```py
>>> from tensorflow import keras
>>> from tensorflow.keras import layers

>>> size = (image_processor.size["height"], image_processor.size["width"])

>>> train_data_augmentation = keras.Sequential(
...     [
...         layers.RandomCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...         layers.RandomFlip("horizontal"),
...         layers.RandomRotation(factor=0.02),
...         layers.RandomZoom(height_factor=0.2, width_factor=0.2),
...     ],
...     name="train_data_augmentation",
... )

>>> val_data_augmentation = keras.Sequential(
...     [
...         layers.CenterCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...     ],
...     name="val_data_augmentation",
... )
```

Next, create functions to apply appropriate transformations to a batch of images, instead of one image at a time.

```py
>>> import numpy as np
>>> import tensorflow as tf
>>> from PIL import Image


>>> def convert_to_tf_tensor(image: Image):
...     np_image = np.array(image)
...     tf_image = tf.convert_to_tensor(np_image)
...     # `expand_dims()` is used to add a batch dimension since
...     # the TF augmentation layers operates on batched inputs.
...     return tf.expand_dims(tf_image, 0)


key: _node_content
string_value:
{"id_": "c5462b7e-dfee-464c-a832-05c895076ece", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "accb8ce4-acc8-452a-9206-4cbe936d0eab", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "330de1b9814df82ad569b3da677a4dd02451dc8bac441ec190bb278a46cd36b8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d4298eb6-fe10-4f5d-9917-561993116a9e", "node_type": "1", "metadata": {}, "hash": "856f897edd6dc468ad9d0b428d3806849118fbed4b8e33cd1a5467c5ab711f96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4510, "end_char_idx": 6558, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<tf>

To avoid overfitting and to make the model more robust, add some data augmentation to the training part of the dataset.
Here we use Keras preprocessing layers to define the transformations for the training data (includes data augmentation),
and transformations for the validation data (only center cropping, resizing and normalizing). You can use `tf.image`or
any other library you prefer.

```py
>>> from tensorflow import keras
>>> from tensorflow.keras import layers

>>> size = (image_processor.size["height"], image_processor.size["width"])

>>> train_data_augmentation = keras.Sequential(
...     [
...         layers.RandomCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...         layers.RandomFlip("horizontal"),
...         layers.RandomRotation(factor=0.02),
...         layers.RandomZoom(height_factor=0.2, width_factor=0.2),
...     ],
...     name="train_data_augmentation",
... )

>>> val_data_augmentation = keras.Sequential(
...     [
...         layers.CenterCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...     ],
...     name="val_data_augmentation",
... )
```

Next, create functions to apply appropriate transformations to a batch of images, instead of one image at a time.

```py
>>> import numpy as np
>>> import tensorflow as tf
>>> from PIL import Image


>>> def convert_to_tf_tensor(image: Image):
...     np_image = np.array(image)
...     tf_image = tf.convert_to_tensor(np_image)
...     # `expand_dims()` is used to add a batch dimension since
...     # the TF augmentation layers operates on batched inputs.
...     return tf.expand_dims(tf_image, 0)


>>> def preprocess_train(example_batch):
...     """Apply train_transforms across a batch."""
...     images = [
...         train_data_augmentation(convert_to_tf_tensor(image.convert("RGB"))) for image in example_batch["image"]
...     ]
...     example_batch["pixel_values"] = [tf.transpose(tf.squeeze(image)) for image in images]
...     return example_batch


key: _node_content
string_value:
{"id_": "d4298eb6-fe10-4f5d-9917-561993116a9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5462b7e-dfee-464c-a832-05c895076ece", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "17e2506a84814dbcb41f78b45100977ffd5d31d3589f976900fc1baaaa8560ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26f39734-cd1b-4e4d-88c8-94a7c42e9f1e", "node_type": "1", "metadata": {}, "hash": "8ebf729f7c3e3d478a005928d544cdf05d8f33e29259247aa51839f57548fbca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5893, "end_char_idx": 9273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def convert_to_tf_tensor(image: Image):
...     np_image = np.array(image)
...     tf_image = tf.convert_to_tensor(np_image)
...     # `expand_dims()` is used to add a batch dimension since
...     # the TF augmentation layers operates on batched inputs.
...     return tf.expand_dims(tf_image, 0)


>>> def preprocess_train(example_batch):
...     """Apply train_transforms across a batch."""
...     images = [
...         train_data_augmentation(convert_to_tf_tensor(image.convert("RGB"))) for image in example_batch["image"]
...     ]
...     example_batch["pixel_values"] = [tf.transpose(tf.squeeze(image)) for image in images]
...     return example_batch


... def preprocess_val(example_batch):
...     """Apply val_transforms across a batch."""
...     images = [
...         val_data_augmentation(convert_to_tf_tensor(image.convert("RGB"))) for image in example_batch["image"]
...     ]
...     example_batch["pixel_values"] = [tf.transpose(tf.squeeze(image)) for image in images]
...     return example_batch
```

Use 🤗 Datasets [`~datasets.Dataset.set_transform`] to apply the transformations on the fly:

```py
food["train"].set_transform(preprocess_train)
food["test"].set_transform(preprocess_val)
```

As a final preprocessing step, create a batch of examples using `DefaultDataCollator`. Unlike other data collators in 🤗 Transformers, the
`DefaultDataCollator` does not apply additional preprocessing, such as padding.

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator(return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load an
evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load
the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you set up your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load ViT with [`AutoModelForImageClassification`]. Specify the number of labels along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForImageClassification, TrainingArguments, Trainer

>>> model = AutoModelForImageClassification.from_pretrained(
...     checkpoint,
...     num_labels=len(labels),
...     id2label=id2label,
...     label2id=label2id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "26f39734-cd1b-4e4d-88c8-94a7c42e9f1e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d4298eb6-fe10-4f5d-9917-561993116a9e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d27782030d9058ebc9ccd6dc7c5d191fc80bde3aa9f393926d4ac60b21e2d1d2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e6c06a88-7033-4c61-8279-8f15d140f945", "node_type": "1", "metadata": {}, "hash": "aad4b2649d3d96efc2568844bc5596a332a52834936a2874b294480a27ab0e45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8180, "end_char_idx": 11434, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you set up your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load ViT with [`AutoModelForImageClassification`]. Specify the number of labels along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForImageClassification, TrainingArguments, Trainer

>>> model = AutoModelForImageClassification.from_pretrained(
...     checkpoint,
...     num_labels=len(labels),
...     id2label=id2label,
...     label2id=label2id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because that'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_food_model",
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=food["train"],
...     eval_dataset=food["test"],
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>

<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pre-trained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.


key: _node_content
string_value:
{"id_": "e6c06a88-7033-4c61-8279-8f15d140f945", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26f39734-cd1b-4e4d-88c8-94a7c42e9f1e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "366c190b600e76aa7bae7ed638deaee7881f42ac9a475858ccf3f57c3d1bf324", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96d4f8df-d57c-410c-b023-cc8a120a1d01", "node_type": "1", "metadata": {}, "hash": "04349a153348402a3382c78e9f4a8686e0ab9bf73f30b160645e7f4168f12582", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9943, "end_char_idx": 11581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_food_model",
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=food["train"],
...     eval_dataset=food["test"],
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>

<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pre-trained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks and use the `fit()` method to run the training.
6. Upload your model to 🤗 Hub to share with the community.


key: _node_content
string_value:
{"id_": "96d4f8df-d57c-410c-b023-cc8a120a1d01", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e6c06a88-7033-4c61-8279-8f15d140f945", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8be00a90a19631ef10fab83bb927bd2ff548cc92b40d05ff6ee68cd79e51dd7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1e4d49ef-b9e8-492c-8d6f-54e2b93d1f12", "node_type": "1", "metadata": {}, "hash": "91f91fabf466cad9594a8578f516903c3aa88d8bb82fac7060a63121cbab67c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9995, "end_char_idx": 13354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


