# embedding_metadata

key: chroma:document
string_value:
273200</td>
      <td>0.445600</td>
      <td>0.167000</td>
      <td>0.346900</td>
      <td>0.160100</td>
      <td>0.387700</td>
      <td>0.249200</td>
      <td>0.392900</td>
    </tr>
    <tr>
      <td>28</td>
      <td>1.049700</td>
      <td>1.346533</td>
      <td>0.277000</td>
      <td>0.552800</td>
      <td>0.252900</td>
      <td>0.147400</td>
      <td>0.240000</td>
      <td>0.527600</td>
      <td>0.280900</td>
      <td>0.420900</td>
      <td>0.444100</td>
      <td>0.255500</td>
      <td>0.424500</td>
      <td>0.711200</td>
      <td>0.530200</td>
      <td>0.646800</td>
      <td>0.277400</td>
      <td>0.441800</td>
      <td>0.170900</td>
      <td>0.346900</td>
      <td>0.156600</td>
      <td>0.389200</td>
      <td>0.249600</td>
      <td>0.396000</td>
    </tr>
    <tr>
      <td>29</td>
      <td>0.993700</td>
      <td>1.346575</td>
      <td>0.277100</td>
      <td>0.554800</td>
      <td>0.252900</td>
      <td>0.148400</td>
      <td>0.239700</td>
      <td>0.523600</td>
      <td>0.278400</td>
      <td>0.420000</td>
      <td>0.443300</td>
      <td>0.256300</td>
      <td>0.424000</td>
      <td>0.705600</td>
      <td>0.529600</td>
      <td>0.647300</td>
      <td>0.273900</td>
      <td>0.439200</td>
      <td>0.174300</td>
      <td>0.348700</td>
      <td>0.157600</td>
      <td>0.386200</td>
      <td>0.250100</td>
      <td>0.395100</td>
    </tr>
    <tr>
      <td>30</td>
      <td>0.993700</td>
      <td>1.346446</td>
      <td>0.277400</td>
      <td>0.554700</td>
      <td>0.252700</td>
      <td>0.147900</td>
      <td>0.240800</td>
      <td>0.523600</td>
      <td>0.278800</td>
      <td>0.420400</td>
      <td>0.443300</td>
      <td>0.


key: _node_content
string_value:
{"id_": "92f15471-ed88-4dc4-930c-f5a8d39c9bbd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3be1cac-cb30-4ddf-9c10-ffebd53de173", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "75cf3b74e03b2f1bbc7c5ad2b5e71d5ffca6cc15caa636cdcaec6b267f83b0e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4036b18f-940a-4455-a59f-5db3aa14ea79", "node_type": "1", "metadata": {}, "hash": "1ef026ef0d3a9ed9437da9b2d6b1a80a19cc4383f0da115e2f1842ad9d53c285", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42807, "end_char_idx": 44245, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
346575</td>
      <td>0.277100</td>
      <td>0.554800</td>
      <td>0.252900</td>
      <td>0.148400</td>
      <td>0.239700</td>
      <td>0.523600</td>
      <td>0.278400</td>
      <td>0.420000</td>
      <td>0.443300</td>
      <td>0.256300</td>
      <td>0.424000</td>
      <td>0.705600</td>
      <td>0.529600</td>
      <td>0.647300</td>
      <td>0.273900</td>
      <td>0.439200</td>
      <td>0.174300</td>
      <td>0.348700</td>
      <td>0.157600</td>
      <td>0.386200</td>
      <td>0.250100</td>
      <td>0.395100</td>
    </tr>
    <tr>
      <td>30</td>
      <td>0.993700</td>
      <td>1.346446</td>
      <td>0.277400</td>
      <td>0.554700</td>
      <td>0.252700</td>
      <td>0.147900</td>
      <td>0.240800</td>
      <td>0.523600</td>
      <td>0.278800</td>
      <td>0.420400</td>
      <td>0.443300</td>
      <td>0.256100</td>
      <td>0.424200</td>
      <td>0.705500</td>
      <td>0.530100</td>
      <td>0.646800</td>
      <td>0.275600</td>
      <td>0.440500</td>
      <td>0.174500</td>
      <td>0.348700</td>
      <td>0.157300</td>
      <td>0.386200</td>
      <td>0.249200</td>
      <td>0.394200</td>
    </tr>
  </tbody>
</table><p>

If you have set `push_to_hub` to `True` in the `training_args`, the training checkpoints are pushed to the
Hugging Face Hub. Upon training completion, push the final model to the Hub as well by calling the [`~transformers.Trainer.push_to_hub`] method.


key: _node_content
string_value:
{"id_": "4036b18f-940a-4455-a59f-5db3aa14ea79", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92f15471-ed88-4dc4-930c-f5a8d39c9bbd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "294c0852c60673f20a356f1c2b0c66012ff79f920b63c9bfdb007db94b486adb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3bba897a-7994-472c-b3f4-2d1bafe612f8", "node_type": "1", "metadata": {}, "hash": "fef76698782cd924dfa6d083953b9935fd0dc238a9ff8043046ec52647f20817", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43263, "end_char_idx": 45324, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
157600</td>
      <td>0.386200</td>
      <td>0.250100</td>
      <td>0.395100</td>
    </tr>
    <tr>
      <td>30</td>
      <td>0.993700</td>
      <td>1.346446</td>
      <td>0.277400</td>
      <td>0.554700</td>
      <td>0.252700</td>
      <td>0.147900</td>
      <td>0.240800</td>
      <td>0.523600</td>
      <td>0.278800</td>
      <td>0.420400</td>
      <td>0.443300</td>
      <td>0.256100</td>
      <td>0.424200</td>
      <td>0.705500</td>
      <td>0.530100</td>
      <td>0.646800</td>
      <td>0.275600</td>
      <td>0.440500</td>
      <td>0.174500</td>
      <td>0.348700</td>
      <td>0.157300</td>
      <td>0.386200</td>
      <td>0.249200</td>
      <td>0.394200</td>
    </tr>
  </tbody>
</table><p>

If you have set `push_to_hub` to `True` in the `training_args`, the training checkpoints are pushed to the
Hugging Face Hub. Upon training completion, push the final model to the Hub as well by calling the [`~transformers.Trainer.push_to_hub`] method.

```py
>>> trainer.push_to_hub()
```

## Evaluate

```py
>>> from pprint import pprint

>>> metrics = trainer.evaluate(eval_dataset=cppe5["test"], metric_key_prefix="test")
>>> pprint(metrics)
{'epoch': 30.0,
  'test_loss': 1.0877351760864258,
  'test_map': 0.4116,
  'test_map_50': 0.741,
  'test_map_75': 0.3663,
  'test_map_Coverall': 0.5937,
  'test_map_Face_Shield': 0.5863,
  'test_map_Gloves': 0.3416,
  'test_map_Goggles': 0.1468,
  'test_map_Mask': 0.3894,
  'test_map_large': 0.5637,
  'test_map_medium': 0.3257,
  'test_map_small': 0.3589,
  'test_mar_1': 0.323,
  'test_mar_10': 0.5237,
  'test_mar_100': 0.5587,
  'test_mar_100_Coverall': 0.6756,
  'test_mar_100_Face_Shield': 0.7294,
  'test_mar_100_Gloves': 0.4721,
  'test_mar_100_Goggles': 0.4125,
  'test_mar_100_Mask': 0.5038,
  'test_mar_large': 0.7283,
  'test_mar_medium': 0.4901,
  'test_mar_small': 0.4469,
  'test_runtime': 1.6526,
  'test_samples_per_second': 17.548,
  'test_steps_per_second': 2.42}
```

These results can be further improved by adjusting the hyperparameters in [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "3bba897a-7994-472c-b3f4-2d1bafe612f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c67b77e2-0310-4df5-8659-cf6d0618167d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "d49b6ae8ff266cc10e090c1ef796434af536b844319b2f94a799eb913f650fc6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4036b18f-940a-4455-a59f-5db3aa14ea79", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/object_detection", "title": "Object detection", "tokens": 16536, "retrieve_doc": false, "source": "transformers"}, "hash": "c01bfcedcc9708bb38549065936edfbcf3b16f93579ee0683ee90102e00dc845", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 45325, "end_char_idx": 47788, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Give it a go!

## Inference

Now that you have finetuned a model, evaluated it, and uploaded it to the Hugging Face Hub, you can use it for inference.

```py
>>> import torch
>>> import requests

>>> from PIL import Image, ImageDraw
>>> from transformers import AutoImageProcessor, AutoModelForObjectDetection

>>> url = "https://images.pexels.com/photos/8413299/pexels-photo-8413299.jpeg?auto=compress&cs=tinysrgb&w=630&h=375&dpr=2"
>>> image = Image.open(requests.get(url, stream=True).raw)
```

Load model and image processor from the Hugging Face Hub (skip to use already trained in this session):
```py
>>> device = "cuda"
>>> model_repo = "qubvel-hf/detr_finetuned_cppe5"

>>> image_processor = AutoImageProcessor.from_pretrained(model_repo)
>>> model = AutoModelForObjectDetection.from_pretrained(model_repo)
>>> model = model.to(device)
```

And detect bounding boxes:

```py

>>> with torch.no_grad():
...     inputs = image_processor(images=[image], return_tensors="pt")
...     outputs = model(**inputs.to(device))
...     target_sizes = torch.tensor([[image.size[1], image.size[0]]])
...     results = image_processor.post_process_object_detection(outputs, threshold=0.3, target_sizes=target_sizes)[0]

>>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
...     box = [round(i, 2) for i in box.tolist()]
...     print(
...         f"Detected {model.config.id2label[label.item()]} with confidence "
...         f"{round(score.item(), 3)} at location {box}"
...     )
Detected Gloves with confidence 0.683 at location [244.58, 124.33, 300.35, 185.13]
Detected Mask with confidence 0.517 at location [143.73, 64.58, 219.57, 125.89]
Detected Gloves with confidence 0.425 at location [179.15, 155.57, 262.4, 226.35]
Detected Coverall with confidence 0.407 at location [307.13, -1.18, 477.82, 318.06]
Detected Coverall with confidence 0.391 at location [68.61, 126.66, 309.03, 318.89]
```

Let's plot the result:

```py
>>> draw = ImageDraw.Draw(image)

>>> for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
...     box = [round(i, 2) for i in box.tolist()]
...     x, y, x2, y2 = tuple(box)
...     draw.rectangle((x, y, x2, y2), outline="red", width=1)
...     draw.text((x, y), model.config.id2label[label.item()], fill="white")

>>> image
```

<div class="flex justify-center">
    <img src="https://i.imgur.com/oDUqD0K.png" alt="Object detection result on a new image"/>
</div>


key: _node_content
string_value:
{"id_": "691da99c-3f64-45c7-9de9-ef70575bbfdc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "827f035b-0e63-4c14-b8c1-1da0fc0c446a", "node_type": "1", "metadata": {}, "hash": "7fad541e01f49fb3c14c95628ab35eaf9915dee7237b3c7b3e8f2e6bf913debd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2505, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Token classification

[[open-in-colab]]

<Youtube id="wVHdVlPScxA"/>

Token classification assigns a label to individual tokens in a sentence. One of the most common token classification tasks is Named Entity Recognition (NER). NER attempts to find a label for each entity in a sentence, such as a person, location, or organization.

This guide will show you how to:

1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [WNUT 17](https://huggingface.co/datasets/wnut_17) dataset to detect new entities.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/token-classification).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate seqeval
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load WNUT 17 dataset

Start by loading the WNUT 17 dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> wnut = load_dataset("wnut_17")
```

Then take a look at an example:

```py
>>> wnut["train"][0]
{'id': '0',
 'ner_tags': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 8, 8, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0],
 'tokens': ['@paulwalk', 'It', "'s", 'the', 'view', 'from', 'where', 'I', "'m", 'living', 'for', 'two', 'weeks', '.', 'Empire', 'State', 'Building', '=', 'ESB', '.', 'Pretty', 'bad', 'storm', 'here', 'last', 'evening', '.']
}
```

Each number in `ner_tags` represents an entity. Convert the numbers to their label names to find out what the entities are:

```py
>>> label_list = wnut["train"].features[f"ner_tags"].feature.names
>>> label_list
[
    "O",
    "B-corporation",
    "I-corporation",
    "B-creative-work",
    "I-creative-work",
    "B-group",
    "I-group",
    "B-location",
    "I-location",
    "B-person",
    "I-person",
    "B-product",
    "I-product",
]
```

The letter that prefixes each `ner_tag` indicates the token position of the entity:

- `B-` indicates the beginning of an entity.
- `I-` indicates a token is contained inside the same entity (for example, the `State` token is a part of an entity like
  `Empire State Building`).
- `0` indicates the token doesn't correspond to any entity.


key: _node_content
string_value:
{"id_": "827f035b-0e63-4c14-b8c1-1da0fc0c446a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "691da99c-3f64-45c7-9de9-ef70575bbfdc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "4ce66b9f8a5303f783742c32261bd29bd6a7778e5aad92469187666fa05c7771", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "08733de2-ad5c-4c3b-96bd-fe351c359e80", "node_type": "1", "metadata": {}, "hash": "e24a3259e936aa3c545d1ab2fdad9f602f66c385b824ec4ef84154749c1c20b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1600, "end_char_idx": 4255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', 'Empire', 'State', 'Building', '=', 'ESB', '.', 'Pretty', 'bad', 'storm', 'here', 'last', 'evening', '.']
}
```

Each number in `ner_tags` represents an entity. Convert the numbers to their label names to find out what the entities are:

```py
>>> label_list = wnut["train"].features[f"ner_tags"].feature.names
>>> label_list
[
    "O",
    "B-corporation",
    "I-corporation",
    "B-creative-work",
    "I-creative-work",
    "B-group",
    "I-group",
    "B-location",
    "I-location",
    "B-person",
    "I-person",
    "B-product",
    "I-product",
]
```

The letter that prefixes each `ner_tag` indicates the token position of the entity:

- `B-` indicates the beginning of an entity.
- `I-` indicates a token is contained inside the same entity (for example, the `State` token is a part of an entity like
  `Empire State Building`).
- `0` indicates the token doesn't correspond to any entity.

## Preprocess

<Youtube id="iY2AZYdZAr0"/>

The next step is to load a DistilBERT tokenizer to preprocess the `tokens` field:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

As you saw in the example `tokens` field above, it looks like the input has already been tokenized. But the input actually hasn't been tokenized yet and you'll need to set `is_split_into_words=True` to tokenize the words into subwords. For example:

```py
>>> example = wnut["train"][0]
>>> tokenized_input = tokenizer(example["tokens"], is_split_into_words=True)
>>> tokens = tokenizer.convert_ids_to_tokens(tokenized_input["input_ids"])
>>> tokens
['[CLS]', '@', 'paul', '##walk', 'it', "'", 's', 'the', 'view', 'from', 'where', 'i', "'", 'm', 'living', 'for', 'two', 'weeks', '.', 'empire', 'state', 'building', '=', 'es', '##b', '.', 'pretty', 'bad', 'storm', 'here', 'last', 'evening', '.', '[SEP]']
```

However, this adds some special tokens `[CLS]` and `[SEP]` and the subword tokenization creates a mismatch between the input and labels. A single word corresponding to a single label may now be split into two subwords. You'll need to realign the tokens and labels by:

1. Mapping all tokens to their corresponding word with the [`word_ids`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.BatchEncoding.word_ids) method.
2. Assigning the label `-100` to the special tokens `[CLS]` and `[SEP]` so they're ignored by the PyTorch loss function (see [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)).
3. Only labeling the first token of a given word. Assign `-100` to other subtokens from the same word.


key: _node_content
string_value:
{"id_": "08733de2-ad5c-4c3b-96bd-fe351c359e80", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "827f035b-0e63-4c14-b8c1-1da0fc0c446a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "27c61547823b6dade56a4b83b332b2e84f6e9837ecaa4036bc851501359a1b35", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2f3809c-9493-42fb-af0c-f2e533ce9c6d", "node_type": "1", "metadata": {}, "hash": "8b40dc6404aa43958b99d6dba236aab721b1780867b9fbaff0852f08e6ba03de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2873, "end_char_idx": 5883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But the input actually hasn't been tokenized yet and you'll need to set `is_split_into_words=True` to tokenize the words into subwords. For example:

```py
>>> example = wnut["train"][0]
>>> tokenized_input = tokenizer(example["tokens"], is_split_into_words=True)
>>> tokens = tokenizer.convert_ids_to_tokens(tokenized_input["input_ids"])
>>> tokens
['[CLS]', '@', 'paul', '##walk', 'it', "'", 's', 'the', 'view', 'from', 'where', 'i', "'", 'm', 'living', 'for', 'two', 'weeks', '.', 'empire', 'state', 'building', '=', 'es', '##b', '.', 'pretty', 'bad', 'storm', 'here', 'last', 'evening', '.', '[SEP]']
```

However, this adds some special tokens `[CLS]` and `[SEP]` and the subword tokenization creates a mismatch between the input and labels. A single word corresponding to a single label may now be split into two subwords. You'll need to realign the tokens and labels by:

1. Mapping all tokens to their corresponding word with the [`word_ids`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.BatchEncoding.word_ids) method.
2. Assigning the label `-100` to the special tokens `[CLS]` and `[SEP]` so they're ignored by the PyTorch loss function (see [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)).
3. Only labeling the first token of a given word. Assign `-100` to other subtokens from the same word.

Here is how you can create a function to realign the tokens and labels, and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def tokenize_and_align_labels(examples):
...     tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)

...     labels = []
...     for i, label in enumerate(examples[f"ner_tags"]):
...         word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
...         previous_word_idx = None
...         label_ids = []
...         for word_idx in word_ids:  # Set the special tokens to -100.
...             if word_idx is None:
...                 label_ids.append(-100)
...             elif word_idx != previous_word_idx:  # Only label the first token of a given word.
...                 label_ids.append(label[word_idx])
...             else:
...                 label_ids.append(-100)
...             previous_word_idx = word_idx
...         labels.append(label_ids)

...     tokenized_inputs["labels"] = labels
...     return tokenized_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.


key: _node_content
string_value:
{"id_": "e2f3809c-9493-42fb-af0c-f2e533ce9c6d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "08733de2-ad5c-4c3b-96bd-fe351c359e80", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "ae7a759a0092c922744e5f5a3cd4cf4bb140daef03781c949842b6117c94eef4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6efde9e8-ae4c-4e95-a872-f827b56372c9", "node_type": "1", "metadata": {}, "hash": "c48f8e218afc02521019a541ca338e311e5413d985d6935bb41b53308401feca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4153, "end_char_idx": 7140, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Only labeling the first token of a given word. Assign `-100` to other subtokens from the same word.

Here is how you can create a function to realign the tokens and labels, and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def tokenize_and_align_labels(examples):
...     tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)

...     labels = []
...     for i, label in enumerate(examples[f"ner_tags"]):
...         word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
...         previous_word_idx = None
...         label_ids = []
...         for word_idx in word_ids:  # Set the special tokens to -100.
...             if word_idx is None:
...                 label_ids.append(-100)
...             elif word_idx != previous_word_idx:  # Only label the first token of a given word.
...                 label_ids.append(label[word_idx])
...             else:
...                 label_ids.append(-100)
...             previous_word_idx = word_idx
...         labels.append(label_ids)

...     tokenized_inputs["labels"] = labels
...     return tokenized_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) framework (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric). Seqeval actually produces several scores: precision, recall, F1, and accuracy.

```py
>>> import evaluate

>>> seqeval = evaluate.load("seqeval")
```

Get the NER labels first, and then create a function that passes your true predictions and true labels to [`~evaluate.EvaluationModule.compute`] to calculate the scores:

```py
>>> import numpy as np

>>> labels = [label_list[i] for i in example[f"ner_tags"]]


key: _node_content
string_value:
{"id_": "6efde9e8-ae4c-4e95-a872-f827b56372c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2f3809c-9493-42fb-af0c-f2e533ce9c6d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "725395a4524e9155d3cd49543a87e29c340894bbf88935f95cc92383a96dca3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f87b2342-4377-41cb-bd3d-a70adfb9b5ff", "node_type": "1", "metadata": {}, "hash": "2785ed4cda887f25cbbfdbf7f11d1469fc8fb6cbc35dc7fd9c5fb56868f7f0ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5454, "end_char_idx": 8034, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_wnut = wnut.map(tokenize_and_align_labels, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorForTokenClassification

>>> data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) framework (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric). Seqeval actually produces several scores: precision, recall, F1, and accuracy.

```py
>>> import evaluate

>>> seqeval = evaluate.load("seqeval")
```

Get the NER labels first, and then create a function that passes your true predictions and true labels to [`~evaluate.EvaluationModule.compute`] to calculate the scores:

```py
>>> import numpy as np

>>> labels = [label_list[i] for i in example[f"ner_tags"]]


>>> def compute_metrics(p):
...     predictions, labels = p
...     predictions = np.argmax(predictions, axis=2)

...     true_predictions = [
...         [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]
...     true_labels = [
...         [label_list[l] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]

...     results = seqeval.compute(predictions=true_predictions, references=true_labels)
...     return {
...         "precision": results["overall_precision"],
...         "recall": results["overall_recall"],
...         "f1": results["overall_f1"],
...         "accuracy": results["overall_accuracy"],
...     }
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.


key: _node_content
string_value:
{"id_": "f87b2342-4377-41cb-bd3d-a70adfb9b5ff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6efde9e8-ae4c-4e95-a872-f827b56372c9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "519ea7b6fd7abbda59d6c6d09cf6639fe2d3698a44a4dd745447b64ea88bf7c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b5891f24-a444-4f5b-b6f4-b30550bade31", "node_type": "1", "metadata": {}, "hash": "361ddae4d20d8a9763a0df4fb87d8f3d209da7e2a76e44dc9857432d305f8b1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6376, "end_char_idx": 9106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [seqeval](https://huggingface.co/spaces/evaluate-metric/seqeval) framework (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric). Seqeval actually produces several scores: precision, recall, F1, and accuracy.

```py
>>> import evaluate

>>> seqeval = evaluate.load("seqeval")
```

Get the NER labels first, and then create a function that passes your true predictions and true labels to [`~evaluate.EvaluationModule.compute`] to calculate the scores:

```py
>>> import numpy as np

>>> labels = [label_list[i] for i in example[f"ner_tags"]]


>>> def compute_metrics(p):
...     predictions, labels = p
...     predictions = np.argmax(predictions, axis=2)

...     true_predictions = [
...         [label_list[p] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]
...     true_labels = [
...         [label_list[l] for (p, l) in zip(prediction, label) if l != -100]
...         for prediction, label in zip(predictions, labels)
...     ]

...     results = seqeval.compute(predictions=true_predictions, references=true_labels)
...     return {
...         "precision": results["overall_precision"],
...         "recall": results["overall_recall"],
...         "f1": results["overall_f1"],
...         "accuracy": results["overall_accuracy"],
...     }
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {
...     0: "O",
...     1: "B-corporation",
...     2: "I-corporation",
...     3: "B-creative-work",
...     4: "I-creative-work",
...     5: "B-group",
...     6: "I-group",
...     7: "B-location",
...     8: "I-location",
...     9: "B-person",
...     10: "I-person",
...     11: "B-product",
...     12: "I-product",
... }
>>> label2id = {
...     "O": 0,
...     "B-corporation": 1,
...     "I-corporation": 2,
...     "B-creative-work": 3,
...     "I-creative-work": 4,
...     "B-group": 5,
...     "I-group": 6,
...     "B-location": 7,
...     "I-location": 8,
...     "B-person": 9,
...     "I-person": 10,
...     "B-product": 11,
...     "I-product": 12,
... }
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now!


key: _node_content
string_value:
{"id_": "b5891f24-a444-4f5b-b6f4-b30550bade31", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f87b2342-4377-41cb-bd3d-a70adfb9b5ff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "aa6683ba18182c25b2cc055d59626c3ba351e1089d431ef5cb3ff9e85823405b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "65581ac0-b114-49c8-bd80-b1d6c811d3e4", "node_type": "1", "metadata": {}, "hash": "d14c6500bb75218c36c18dc1c412f9c3e9419f26650964188997e7522391b8ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8036, "end_char_idx": 10113, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {
...     0: "O",
...     1: "B-corporation",
...     2: "I-corporation",
...     3: "B-creative-work",
...     4: "I-creative-work",
...     5: "B-group",
...     6: "I-group",
...     7: "B-location",
...     8: "I-location",
...     9: "B-person",
...     10: "I-person",
...     11: "B-product",
...     12: "I-product",
... }
>>> label2id = {
...     "O": 0,
...     "B-corporation": 1,
...     "I-corporation": 2,
...     "B-creative-work": 3,
...     "I-creative-work": 4,
...     "B-group": 5,
...     "I-group": 6,
...     "B-location": 7,
...     "I-location": 8,
...     "B-person": 9,
...     "I-person": 10,
...     "B-product": 11,
...     "I-product": 12,
... }
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

>>> model = AutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the seqeval scores and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "65581ac0-b114-49c8-bd80-b1d6c811d3e4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b5891f24-a444-4f5b-b6f4-b30550bade31", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "0280928dc6375ed54195f5e586abad9f19bdd956c0fddefa486a5009e7614840", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b62ea455-6a06-4089-bca3-89bee4790041", "node_type": "1", "metadata": {}, "hash": "884fdc423f1ee4a412f40282735107c2af27bc154c7ad28ac3141014acd96396", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9052, "end_char_idx": 11147, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

>>> model = AutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the seqeval scores and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_wnut_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_wnut["train"],
...     eval_dataset=tokenized_wnut["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "b62ea455-6a06-4089-bca3-89bee4790041", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65581ac0-b114-49c8-bd80-b1d6c811d3e4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "1e3540831ef652f7e28c47514b8ae6dde357e23e02a614c4c4819e7113757a7e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82a6dd4b-3511-4537-8d78-45a672761ef5", "node_type": "1", "metadata": {}, "hash": "e02c7d35bf0351340539f9131f6e3d953747f6cec2f229514f303db3b2a22b5c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9604, "end_char_idx": 12620, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the seqeval scores and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_wnut_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_wnut["train"],
...     eval_dataset=tokenized_wnut["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 3
>>> num_train_steps = (len(tokenized_wnut["train"]) // batch_size) * num_train_epochs
>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=2e-5,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=0.01,
...     num_warmup_steps=0,
... )
```

Then you can load DistilBERT with [`TFAutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_wnut["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_wnut["validation"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method).


key: _node_content
string_value:
{"id_": "82a6dd4b-3511-4537-8d78-45a672761ef5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b62ea455-6a06-4089-bca3-89bee4790041", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "8f7b9e3ebc53bd897cc806d4985a52f9eae694c152746df6f358dca135c247ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87783408-116d-44b3-8676-77022bd716fd", "node_type": "1", "metadata": {}, "hash": "567448bec4c7d17bccd228fbf7f802103157af75729277c11b04a956639fbdb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11149, "end_char_idx": 14184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 3
>>> num_train_steps = (len(tokenized_wnut["train"]) // batch_size) * num_train_epochs
>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=2e-5,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=0.01,
...     num_warmup_steps=0,
... )
```

Then you can load DistilBERT with [`TFAutoModelForTokenClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=13, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_wnut["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_wnut["validation"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the seqeval scores from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_wnut_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!


key: _node_content
string_value:
{"id_": "87783408-116d-44b3-8676-77022bd716fd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82a6dd4b-3511-4537-8d78-45a672761ef5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "d6e9a66da3063064646fd7581ad2822293186d82314458af983ec75bcb306465", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b1e6bf2e-fbbd-489c-8254-1b83eea3a9f7", "node_type": "1", "metadata": {}, "hash": "43649f7ea7aa46f7b0026a055f23fdf5a41ddb27921fa2d7f9bbbb0eb0c58095", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12621, "end_char_idx": 15602, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the seqeval scores from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_wnut_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for token classification, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "The Golden State Warriors are an American professional basketball team based in San Francisco."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for NER with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("ner", model="stevhliu/my_awesome_wnut_model")
>>> classifier(text)
[{'entity': 'B-location',
  'score': 0.42658573,
  'index': 2,
  'word': 'golden',
  'start': 4,
  'end': 10},
 {'entity': 'I-location',
  'score': 0.35856336,
  'index': 3,
  'word': 'state',
  'start': 11,
  'end': 16},
 {'entity': 'B-group',
  'score': 0.3064001,
  'index': 4,
  'word': 'warriors',
  'start': 17,
  'end': 25},
 {'entity': 'B-location',
  'score': 0.65523505,
  'index': 13,
  'word': 'san',
  'start': 80,


key: _node_content
string_value:
{"id_": "b1e6bf2e-fbbd-489c-8254-1b83eea3a9f7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87783408-116d-44b3-8676-77022bd716fd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "e4e37799b293f162c217dabdc0c9f6ff4e485a0aa73d11904590aaf30a6e12ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "405c6bc8-de41-4978-b96d-f03fc885f35c", "node_type": "1", "metadata": {}, "hash": "e2b9c89db616ded8c9a9d885c31b8597cb656a91a069386591f1e5554b823a0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "The Golden State Warriors are an American professional basketball team based in San Francisco."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for NER with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("ner", model="stevhliu/my_awesome_wnut_model")
>>> classifier(text)
[{'entity': 'B-location',
  'score': 0.42658573,
  'index': 2,
  'word': 'golden',
  'start': 4,
  'end': 10},
 {'entity': 'I-location',
  'score': 0.35856336,
  'index': 3,
  'word': 'state',
  'start': 11,
  'end': 16},
 {'entity': 'B-group',
  'score': 0.3064001,
  'index': 4,
  'word': 'warriors',
  'start': 17,
  'end': 25},
 {'entity': 'B-location',
  'score': 0.65523505,
  'index': 13,
  'word': 'san',
  'start': 80,
  'end': 83},
 {'entity': 'B-location',
  'score': 0.4668663,
  'index': 14,
  'word': 'francisco',
  'start': 84,
  'end': 93}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> with torch.no_grad():
.     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predictions = torch.argmax(logits, dim=2)
>>> predicted_token_class = [model.config.id2label[t.item()] for t in predictions[0]]
>>> predicted_token_class
['O',
 'O',
 'B-location',
 'I-location',
 'B-group',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'B-location',
 'B-location',
 'O',
 'O']
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text,


key: _node_content
string_value:
{"id_": "405c6bc8-de41-4978-b96d-f03fc885f35c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e9cfbf-26cb-4967-aec7-5a0879f9fe44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "2e558a5bdc01661668c830be29b792690caf3fba9f44f2ddca71e69822f565f3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b1e6bf2e-fbbd-489c-8254-1b83eea3a9f7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/token_classification", "title": "Token classification", "tokens": 4795, "retrieve_doc": true, "source": "transformers"}, "hash": "392af0077eb869393d0ecb290bc8bc2549b7d02c8bec451b12f7235344573c1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4668663,
  'index': 14,
  'word': 'francisco',
  'start': 84,
  'end': 93}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForTokenClassification

>>> model = AutoModelForTokenClassification.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> with torch.no_grad():
.     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predictions = torch.argmax(logits, dim=2)
>>> predicted_token_class = [model.config.id2label[t.item()] for t in predictions[0]]
>>> predicted_token_class
['O',
 'O',
 'B-location',
 'I-location',
 'B-group',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'B-location',
 'B-location',
 'O',
 'O']
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> inputs = tokenizer(text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForTokenClassification

>>> model = TFAutoModelForTokenClassification.from_pretrained("stevhliu/my_awesome_wnut_model")
>>> logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_token_class_ids = tf.math.argmax(logits, axis=-1)
>>> predicted_token_class = [model.config.id2label[t] for t in predicted_token_class_ids[0].numpy().tolist()]
>>> predicted_token_class
['O',
 'O',
 'B-location',
 'I-location',
 'B-group',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'O',
 'B-location',
 'B-location',
 'O',
 'O']
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "823f65b7-4626-4069-8aa4-b52412f800f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97ac2680-efec-4716-942e-16e0b7994ba1", "node_type": "1", "metadata": {}, "hash": "c8f70e656c6b7ff5b593c66ace565599914eb6283fcc8a714c33a92590bfdcab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text classification

[[open-in-colab]]

<Youtube id="leNG9fN9FQU"/>

Text classification is a common NLP task that assigns a label or class to text. Some of the largest companies run text classification in production for a wide range of practical applications. One of the most popular forms of text classification is sentiment analysis, which assigns a label like 🙂 positive, 🙁 negative, or 😐 neutral to a sequence of text.

This guide will show you how to:

1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [IMDb](https://huggingface.co/datasets/imdb) dataset to determine whether a movie review is positive or negative.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/text-classification).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate accelerate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load IMDb dataset

Start by loading the IMDb dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> imdb = load_dataset("imdb")
```

Then take a look at an example:

```py
>>> imdb["test"][0]
{
    "label": 0,
    "text": "I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi' setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV. It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It's really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it's rubbish as they have to always say \"Gene Roddenberry's Earth...\" otherwise people would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.",
}
```

There are two fields in this dataset:

- `text`: the movie review text.
- `label`: a value that is either `0` for a negative review or `1` for a positive review.


key: _node_content
string_value:
{"id_": "97ac2680-efec-4716-942e-16e0b7994ba1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "823f65b7-4626-4069-8aa4-b52412f800f8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "16d3e10353f9a0eb2f06e33f1768f8dbe4123ba3dcbb484dfc1531a2a4b3b2d6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df3c86e4-85f3-4a05-b9c9-761b6401a8b9", "node_type": "1", "metadata": {}, "hash": "7ee99d2dff91ce91cd085ce898a4341ab4217ee25a1d9fbcfa28f5fc107c9594", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1573, "end_char_idx": 4641, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi' setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV. It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It's really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it's rubbish as they have to always say \"Gene Roddenberry's Earth...\" otherwise people would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.",
}
```

There are two fields in this dataset:

- `text`: the movie review text.
- `label`: a value that is either `0` for a negative review or `1` for a positive review.

## Preprocess

The next step is to load a DistilBERT tokenizer to preprocess the `text` field:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a preprocessing function to tokenize `text` and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def preprocess_function(examples):
...     return tokenizer(examples["text"], truncation=True)
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once:

```py
tokenized_imdb = imdb.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library.


key: _node_content
string_value:
{"id_": "df3c86e4-85f3-4a05-b9c9-761b6401a8b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "97ac2680-efec-4716-942e-16e0b7994ba1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "111df497e755d206da1c0ba0f35896255d027ae0b83c6a87f514d37ed78bef5a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b8a1f960-8783-4747-b752-a65d8154c012", "node_type": "1", "metadata": {}, "hash": "b77769ff2fd9e32b51471f005094ed1162e8a4b968124abc65fbd7b02a882965", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2992, "end_char_idx": 5904, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `label`: a value that is either `0` for a negative review or `1` for a positive review.

## Preprocess

The next step is to load a DistilBERT tokenizer to preprocess the `text` field:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a preprocessing function to tokenize `text` and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def preprocess_function(examples):
...     return tokenizer(examples["text"], truncation=True)
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once:

```py
tokenized_imdb = imdb.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
```
</pt>
<tf>
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {0: "NEGATIVE", 1: "POSITIVE"}
>>> label2id = {"NEGATIVE": 0, "POSITIVE": 1}
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now!


key: _node_content
string_value:
{"id_": "b8a1f960-8783-4747-b752-a65d8154c012", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df3c86e4-85f3-4a05-b9c9-761b6401a8b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "9eaa0ed9a1973c01fcbc4c658becb790837f754fcb4bb8e6393e061b256daace", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8bfb1e30-925f-4bb1-9a08-02f738729675", "node_type": "1", "metadata": {}, "hash": "49ca7d701c0816725ca87ef6d19f2889961420d973a7cf738b421786b87cdb82", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4526, "end_char_idx": 7682, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {0: "NEGATIVE", 1: "POSITIVE"}
>>> label2id = {"NEGATIVE": 0, "POSITIVE": 1}
```

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

>>> model = AutoModelForSequenceClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

<Tip>

[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it.


key: _node_content
string_value:
{"id_": "8bfb1e30-925f-4bb1-9a08-02f738729675", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b8a1f960-8783-4747-b752-a65d8154c012", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "3febbcfe3771c4e2b2d0b5cc741e2e985b13e00ca955795339a8125d37904448", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d80a1b8-edd4-4b42-b0a5-4097dc0ea872", "node_type": "1", "metadata": {}, "hash": "54be051ea1b4f8569ed1e9459e02902f63ba0eba2f56698fd62f4a4bf34693e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6347, "end_char_idx": 8106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

<Tip>

[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it. In this case, you don't need to specify a data collator explicitly.

</Tip>

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "4d80a1b8-edd4-4b42-b0a5-4097dc0ea872", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8bfb1e30-925f-4bb1-9a08-02f738729675", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfb3bbbde2bd61ef9914b4c23003226075eb427f510d9e3cf745e0c215af9c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b29d11d-5996-4a8e-8c90-a2d7f2e0cbd2", "node_type": "1", "metadata": {}, "hash": "7a50748331295bcd0994e80d4a29de9a67d331131de0ae96bcc3673319e535f3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6625, "end_char_idx": 9807, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_model",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=2,
...     weight_decay=0.01,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

<Tip>

[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it. In this case, you don't need to specify a data collator explicitly.

</Tip>

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer
>>> import tensorflow as tf

>>> batch_size = 16
>>> num_epochs = 5
>>> batches_per_epoch = len(tokenized_imdb["train"]) // batch_size
>>> total_train_steps = int(batches_per_epoch * num_epochs)
>>> optimizer, schedule = create_optimizer(init_lr=2e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load DistilBERT with [`TFAutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained(
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_imdb["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_imdb["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!


key: _node_content
string_value:
{"id_": "4b29d11d-5996-4a8e-8c90-a2d7f2e0cbd2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d80a1b8-edd4-4b42-b0a5-4097dc0ea872", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "c8b452214b7304ee7d4d2ab296e72c7ded025dc2b0d82987fd670d2627d800cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e624e04-acdd-4b1d-8520-41525e89513c", "node_type": "1", "metadata": {}, "hash": "0a2b117928ab9b5ba6db604303ff2a0a90432669fc21c51096ff8ba2f8f9ecf9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9581, "end_char_idx": 11945, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for text classification, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`].


key: _node_content
string_value:
{"id_": "4e624e04-acdd-4b1d-8520-41525e89513c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ba6ffc6-fc8f-4c02-ab78-e17886da84ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "0b2c612180b4f3f79bc906ce6f7c0ee445069a6b8280aa7a205b282b565b5367", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b29d11d-5996-4a8e-8c90-a2d7f2e0cbd2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/sequence_classification", "title": "Text classification", "tokens": 3451, "retrieve_doc": true, "source": "transformers"}, "hash": "b2493ecf135965e1a9f4db1bbc5b04dc4c1a04b60256078834767fd449b4b681", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10744, "end_char_idx": 13871, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for text classification, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for sentiment analysis with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model")
>>> classifier(text)
[{'label': 'POSITIVE', 'score': 0.9994940757751465}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("stevhliu/my_awesome_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_class_id = logits.argmax().item()
>>> model.config.id2label[predicted_class_id]
'POSITIVE'
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_model")
>>> inputs = tokenizer(text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained("stevhliu/my_awesome_model")
>>> logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
>>> model.config.id2label[predicted_class_id]
'POSITIVE'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "4f4270bd-a3b4-4d57-b745-fe6871e92921", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "53e55709-8f4a-486b-bf7f-2cf852bdf653", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "hash": "00ef6ba89717fd17365ab61ba0f513f1ea2e522b0fca49368251caaefa8f1614", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "78b5ba2d-472f-4217-9d03-7861ea1173d9", "node_type": "1", "metadata": {}, "hash": "f4101d13ac743e5a7ba8b06700dbae65724d23051cafcc31edef369b6d97a6e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3265, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Knowledge Distillation for Computer Vision

[[open-in-colab]]

Knowledge distillation is a technique used to transfer knowledge from a larger, more complex model (teacher) to a smaller, simpler model (student). To distill knowledge from one model to another, we take a pre-trained teacher model trained on a certain task (image classification for this case) and randomly initialize a student model to be trained on image classification. Next, we train the student model to minimize the difference between it's outputs and the teacher's outputs, thus making it mimic the behavior. It was first introduced in [Distilling the Knowledge in a Neural Network by Hinton et al](https://arxiv.org/abs/1503.02531). In this guide, we will do task-specific knowledge distillation. We will use the [beans dataset](https://huggingface.co/datasets/beans) for this.

This guide demonstrates how you can distill a [fine-tuned ViT model](https://huggingface.co/merve/vit-mobilenet-beans-224) (teacher model) to a [MobileNet](https://huggingface.co/google/mobilenet_v2_1.4_224) (student model) using the [Trainer API](https://huggingface.co/docs/transformers/en/main_classes/trainer#trainer) of 🤗 Transformers. 

Let's install the libraries needed for distillation and evaluating the process. 

```bash
pip install transformers datasets accelerate tensorboard evaluate --upgrade
```

In this example, we are using the `merve/beans-vit-224` model as teacher model. It's an image classification model, based on `google/vit-base-patch16-224-in21k` fine-tuned on beans dataset. We will distill this model to a randomly initialized MobileNetV2.

We will now load the dataset. 

```python
from datasets import load_dataset

dataset = load_dataset("beans")
```

We can use an image processor from either of the models, as in this case they return the same output with same resolution. We will use the `map()` method of `dataset` to apply the preprocessing to every split of the dataset. 

```python
from transformers import AutoImageProcessor
teacher_processor = AutoImageProcessor.from_pretrained("merve/beans-vit-224")

def process(examples):
    processed_inputs = teacher_processor(examples["image"])
    return processed_inputs

processed_datasets = dataset.map(process, batched=True)
```

Essentially, we want the student model (a randomly initialized MobileNet) to mimic the teacher model (fine-tuned vision transformer). To achieve this, we first get the logits output from the teacher and the student. Then, we divide each of them by the parameter `temperature` which controls the importance of each soft target. A parameter called `lambda` weighs the importance of the distillation loss. In this example, we will use `temperature=5` and `lambda=0.5`. We will use the Kullback-Leibler Divergence loss to compute the divergence between the student and teacher. Given two data P and Q, KL Divergence explains how much extra information we need to represent P using Q. If two are identical, their KL divergence is zero, as there's no other information needed to explain P from Q. Thus, in the context of knowledge distillation, KL divergence is useful.


```python
from transformers import TrainingArguments, Trainer
import torch
import torch.nn as nn
import torch.nn.functional as F


key: _node_content
string_value:
{"id_": "78b5ba2d-472f-4217-9d03-7861ea1173d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "53e55709-8f4a-486b-bf7f-2cf852bdf653", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "hash": "00ef6ba89717fd17365ab61ba0f513f1ea2e522b0fca49368251caaefa8f1614", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f4270bd-a3b4-4d57-b745-fe6871e92921", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "hash": "c95b0726a6bf8119ac2749e2a318018cfa190d4078363d8b849ee24301a87bc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fdd9730f-6dc5-4395-b4e4-7669cec69a54", "node_type": "1", "metadata": {}, "hash": "ca5d245df04c246f84d8905fd1b793d20047ad56d69ec53d7cfd79e0310a923f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3137, "end_char_idx": 6506, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import TrainingArguments, Trainer
import torch
import torch.nn as nn
import torch.nn.functional as F


class ImageDistilTrainer(Trainer):
    def __init__(self, teacher_model=None, student_model=None, temperature=None, lambda_param=None,  *args, **kwargs):
        super().__init__(model=student_model, *args, **kwargs)
        self.teacher = teacher_model
        self.student = student_model
        self.loss_function = nn.KLDivLoss(reduction="batchmean")
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        self.teacher.to(device)
        self.teacher.eval()
        self.temperature = temperature
        self.lambda_param = lambda_param

    def compute_loss(self, student, inputs, return_outputs=False):
        student_output = self.student(**inputs)

        with torch.no_grad():
          teacher_output = self.teacher(**inputs)

        # Compute soft targets for teacher and student
        soft_teacher = F.softmax(teacher_output.logits / self.temperature, dim=-1)
        soft_student = F.log_softmax(student_output.logits / self.temperature, dim=-1)

        # Compute the loss
        distillation_loss = self.loss_function(soft_student, soft_teacher) * (self.temperature ** 2)

        # Compute the true label loss
        student_target_loss = student_output.loss

        # Calculate final loss
        loss = (1. - self.lambda_param) * student_target_loss + self.lambda_param * distillation_loss
        return (loss, student_output) if return_outputs else loss
```

We will now login to Hugging Face Hub so we can push our model to the Hugging Face Hub through the `Trainer`. 

```python
from huggingface_hub import notebook_login

notebook_login()
```

Let's set the `TrainingArguments`, the teacher model and the student model. 

```python
from transformers import AutoModelForImageClassification, MobileNetV2Config, MobileNetV2ForImageClassification

training_args = TrainingArguments(
    output_dir="my-awesome-model",
    num_train_epochs=30,
    fp16=True,
    logging_dir=f"{repo_name}/logs",
    logging_strategy="epoch",
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    report_to="tensorboard",
    push_to_hub=True,
    hub_strategy="every_save",
    hub_model_id=repo_name,
    )

num_labels = len(processed_datasets["train"].features["labels"].names)

# initialize models
teacher_model = AutoModelForImageClassification.from_pretrained(
    "merve/beans-vit-224",
    num_labels=num_labels,
    ignore_mismatched_sizes=True
)

# training MobileNetV2 from scratch
student_config = MobileNetV2Config()
student_config.num_labels = num_labels
student_model = MobileNetV2ForImageClassification(student_config)
```

We can use `compute_metrics` function to evaluate our model on the test set. This function will be used during the training process to compute the `accuracy` & `f1` of our model.

```python
import evaluate
import numpy as np

accuracy = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    acc = accuracy.compute(references=labels, predictions=np.argmax(predictions, axis=1))
    return {"accuracy": acc["accuracy"]}
```

Let's initialize the `Trainer` with the training arguments we defined. We will also initialize our data collator.


key: _node_content
string_value:
{"id_": "fdd9730f-6dc5-4395-b4e4-7669cec69a54", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "53e55709-8f4a-486b-bf7f-2cf852bdf653", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "hash": "00ef6ba89717fd17365ab61ba0f513f1ea2e522b0fca49368251caaefa8f1614", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78b5ba2d-472f-4217-9d03-7861ea1173d9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/knowledge_distillation_for_image_classification", "title": "Knowledge Distillation for Computer Vision", "tokens": 1908, "retrieve_doc": true, "source": "transformers"}, "hash": "db04bb44c00f3d21fd1b0d5568e8bed7d15e3c9048c08289282417f14d801a9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4800, "end_char_idx": 7801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from huggingface_hub import notebook_login

notebook_login()
```

Let's set the `TrainingArguments`, the teacher model and the student model. 

```python
from transformers import AutoModelForImageClassification, MobileNetV2Config, MobileNetV2ForImageClassification

training_args = TrainingArguments(
    output_dir="my-awesome-model",
    num_train_epochs=30,
    fp16=True,
    logging_dir=f"{repo_name}/logs",
    logging_strategy="epoch",
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    report_to="tensorboard",
    push_to_hub=True,
    hub_strategy="every_save",
    hub_model_id=repo_name,
    )

num_labels = len(processed_datasets["train"].features["labels"].names)

# initialize models
teacher_model = AutoModelForImageClassification.from_pretrained(
    "merve/beans-vit-224",
    num_labels=num_labels,
    ignore_mismatched_sizes=True
)

# training MobileNetV2 from scratch
student_config = MobileNetV2Config()
student_config.num_labels = num_labels
student_model = MobileNetV2ForImageClassification(student_config)
```

We can use `compute_metrics` function to evaluate our model on the test set. This function will be used during the training process to compute the `accuracy` & `f1` of our model.

```python
import evaluate
import numpy as np

accuracy = evaluate.load("accuracy")

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    acc = accuracy.compute(references=labels, predictions=np.argmax(predictions, axis=1))
    return {"accuracy": acc["accuracy"]}
```

Let's initialize the `Trainer` with the training arguments we defined. We will also initialize our data collator.

```python
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator()
trainer = ImageDistilTrainer(
    student_model=student_model,
    teacher_model=teacher_model,
    training_args=training_args,
    train_dataset=processed_datasets["train"],
    eval_dataset=processed_datasets["validation"],
    data_collator=data_collator,
    tokenizer=teacher_processor,
    compute_metrics=compute_metrics,
    temperature=5,
    lambda_param=0.5
)
```

We can now train our model.

```python
trainer.train()
```

We can evaluate the model on the test set.

```python
trainer.evaluate(processed_datasets["test"])
```

On test set, our model reaches 72 percent accuracy. To have a sanity check over efficiency of distillation, we also trained MobileNet on the beans dataset from scratch with the same hyperparameters and observed 63 percent accuracy on the test set. We invite the readers to try different pre-trained teacher models, student architectures, distillation parameters and report their findings. The training logs and checkpoints for distilled model can be found in [this repository](https://huggingface.co/merve/vit-mobilenet-beans-224), and MobileNetV2 trained from scratch can be found in this [repository](https://huggingface.co/merve/resnet-mobilenet-beans-5).


key: _node_content
string_value:
{"id_": "83e1c917-8506-42a8-bdfe-c922cfef912f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ac1f2d2-07a8-441b-b0b4-d884bb01cb77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "hash": "ada39cb91a9e953c778492b4acff885c647402f9324c689d38146231867a72fa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "34615b8d-952c-4959-b1ef-5627f6b20c22", "node_type": "1", "metadata": {}, "hash": "1767be3d4528c63af3a20d0e2272351bb3b493bf92d4441a2cfd515980d2c582", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Zero-shot image classification

[[open-in-colab]]

Zero-shot image classification is a task that involves classifying images into different categories using a model that was
not explicitly trained on data containing labeled examples from those specific categories.

Traditionally, image classification requires training a model on a specific set of labeled images, and this model learns to
"map" certain image features to labels. When there's a need to use such model for a classification task that introduces a
new set of labels, fine-tuning is required to "recalibrate" the model.

In contrast, zero-shot or open vocabulary image classification models are typically multi-modal models that have been trained on a large
dataset of images and associated descriptions. These models learn aligned vision-language representations that can be used for many downstream tasks including zero-shot image classification.

This is a more flexible approach to image classification that allows models to generalize to new and unseen categories
without the need for additional training data and enables users to query images with free-form text descriptions of their target objects .

In this guide you'll learn how to:

* create a zero-shot image classification pipeline
* run zero-shot image classification inference by hand

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q "transformers[torch]" pillow
```

## Zero-shot image classification pipeline

The simplest way to try out inference with a model supporting zero-shot image classification is to use the corresponding [`pipeline`].
Instantiate a pipeline from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=zero-shot-image-classification&sort=downloads):

```python
>>> from transformers import pipeline

>>> checkpoint = "openai/clip-vit-large-patch14"
>>> detector = pipeline(model=checkpoint, task="zero-shot-image-classification")
```

Next, choose an image you'd like to classify.

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/g8oS8-82DxI/download?ixid=MnwxMjA3fDB8MXx0b3BpY3x8SnBnNktpZGwtSGt8fHx8fDJ8fDE2NzgxMDYwODc&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/owl.jpg" alt="Photo of an owl"/>
</div>

Pass the image and the candidate object labels to the pipeline. Here we pass the image directly; other suitable options
include a local path to an image or an image url.
The candidate labels can be simple words like in this example, or more descriptive.

```py
>>> predictions = detector(image, candidate_labels=["fox", "bear", "seagull", "owl"])
>>> predictions
[{'score': 0.9996670484542847, 'label': 'owl'},
 {'score': 0.000199399160919711, 'label': 'seagull'},
 {'score': 7.392891711788252e-05, 'label': 'fox'},
 {'score': 5.96074532950297e-05, 'label': 'bear'}]
```

## Zero-shot image classification by hand

Now that you've seen how to use the zero-shot image classification pipeline, let's take a look how you can run zero-shot
image classification manually.


key: _node_content
string_value:
{"id_": "34615b8d-952c-4959-b1ef-5627f6b20c22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ac1f2d2-07a8-441b-b0b4-d884bb01cb77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "hash": "ada39cb91a9e953c778492b4acff885c647402f9324c689d38146231867a72fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83e1c917-8506-42a8-bdfe-c922cfef912f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "hash": "bd975a3068313dae4e78e7b7c967d2fe49c9bb8e22b8074de1a81cf42944325e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7e3ea57-d3bb-459e-b677-b629a111615e", "node_type": "1", "metadata": {}, "hash": "50b3783ae1d12f9b73763d3c05c29bf00ed838cff6ec905a114117423236e6b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2020, "end_char_idx": 4604, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/g8oS8-82DxI/download?ixid=MnwxMjA3fDB8MXx0b3BpY3x8SnBnNktpZGwtSGt8fHx8fDJ8fDE2NzgxMDYwODc&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/owl.jpg" alt="Photo of an owl"/>
</div>

Pass the image and the candidate object labels to the pipeline. Here we pass the image directly; other suitable options
include a local path to an image or an image url.
The candidate labels can be simple words like in this example, or more descriptive.

```py
>>> predictions = detector(image, candidate_labels=["fox", "bear", "seagull", "owl"])
>>> predictions
[{'score': 0.9996670484542847, 'label': 'owl'},
 {'score': 0.000199399160919711, 'label': 'seagull'},
 {'score': 7.392891711788252e-05, 'label': 'fox'},
 {'score': 5.96074532950297e-05, 'label': 'bear'}]
```

## Zero-shot image classification by hand

Now that you've seen how to use the zero-shot image classification pipeline, let's take a look how you can run zero-shot
image classification manually.

Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=zero-shot-image-classification&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoProcessor, AutoModelForZeroShotImageClassification

>>> model = AutoModelForZeroShotImageClassification.from_pretrained(checkpoint)
>>> processor = AutoProcessor.from_pretrained(checkpoint)
```

Let's take a different image to switch things up.

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/xBRQfR2bqNI/download?ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjc4Mzg4ODEx&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg" alt="Photo of a car"/>
</div>

Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the
image for the model by resizing and normalizing it, and a tokenizer that takes care of the text inputs.

```py
>>> candidate_labels = ["tree", "car", "bike", "cat"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.'


key: _node_content
string_value:
{"id_": "f7e3ea57-d3bb-459e-b677-b629a111615e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ac1f2d2-07a8-441b-b0b4-d884bb01cb77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "hash": "ada39cb91a9e953c778492b4acff885c647402f9324c689d38146231867a72fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "34615b8d-952c-4959-b1ef-5627f6b20c22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification", "title": "Zero-shot image classification", "tokens": 1461, "retrieve_doc": true, "source": "transformers"}, "hash": "640e4ade939a089bbe67e52ad2f99662109ffb72b241e3772da26bcf42b88a32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3250, "end_char_idx": 5350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=zero-shot-image-classification&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoProcessor, AutoModelForZeroShotImageClassification

>>> model = AutoModelForZeroShotImageClassification.from_pretrained(checkpoint)
>>> processor = AutoProcessor.from_pretrained(checkpoint)
```

Let's take a different image to switch things up.

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://unsplash.com/photos/xBRQfR2bqNI/download?ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjc4Mzg4ODEx&force=true&w=640"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg" alt="Photo of a car"/>
</div>

Use the processor to prepare the inputs for the model. The processor combines an image processor that prepares the
image for the model by resizing and normalizing it, and a tokenizer that takes care of the text inputs.

```py
>>> candidate_labels = ["tree", "car", "bike", "cat"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]
>>> inputs = processor(images=image, text=candidate_labels, return_tensors="pt", padding=True)
```

Pass the inputs through the model, and post-process the results:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits = outputs.logits_per_image[0]
>>> probs = logits.softmax(dim=-1).numpy()
>>> scores = probs.tolist()

>>> result = [
...     {"score": score, "label": candidate_label}
...     for score, candidate_label in sorted(zip(probs, candidate_labels), key=lambda x: -x[0])
... ]

>>> result
[{'score': 0.998572, 'label': 'car'},
 {'score': 0.0010570387, 'label': 'bike'},
 {'score': 0.0003393686, 'label': 'tree'},
 {'score': 3.1572064e-05, 'label': 'cat'}]
```


key: _node_content
string_value:
{"id_": "1610bb4d-e366-4424-af9c-43c772a8ca45", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c72d755f-1c90-4e31-a34e-c3beaf6043ff", "node_type": "1", "metadata": {}, "hash": "c57eb3e8f8f601c7c77ae445c2f7e1a83ae1b629cfa57d4d8079823b0f06dd8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Visual Question Answering

[[open-in-colab]]

Visual Question Answering (VQA) is the task of answering open-ended questions based on an image. 
The input to models supporting this task is typically a combination of an image and a question, and the output is an 
answer expressed in natural language.

Some noteworthy use case examples for VQA include:
* Accessibility applications for visually impaired individuals.
* Education: posing questions about visual materials presented in lectures or textbooks. VQA can also be utilized in interactive museum exhibits or historical sites.
* Customer service and e-commerce: VQA can enhance user experience by letting users ask questions about products. 
* Image retrieval: VQA models can be used to retrieve images with specific characteristics. For example, the user can ask "Is there a dog?" to find all images with dogs from a set of images.

In this guide you'll learn how to:

- Fine-tune a classification VQA model, specifically [ViLT](../model_doc/vilt), on the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa).
- Use your fine-tuned ViLT for inference.
- Run zero-shot VQA inference with a generative model, like BLIP-2.

## Fine-tuning ViLT

ViLT model incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design for 
Vision-and-Language Pre-training (VLP). This model can be used for several downstream tasks. For the VQA task, a classifier 
head is placed on top (a linear layer on top of the final hidden state of the `[CLS]` token) and randomly initialized. 
Visual Question Answering is thus treated as a **classification problem**.

More recent models, such as BLIP, BLIP-2, and InstructBLIP, treat VQA as a generative task. Later in this guide we 
illustrate how to use them for zero-shot VQA inference. 

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q transformers datasets
```

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define the model checkpoint as a global variable.

```py
>>> model_checkpoint = "dandelin/vilt-b32-mlm"
```

## Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering `Graphcore/vqa` dataset. 
You can find the full dataset on [🤗 Hub](https://huggingface.co/datasets/Graphcore/vqa).

As an alternative to the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa), you can download the 
same data manually from the official [VQA dataset page](https://visualqa.org/download.html). If you prefer to follow the 
tutorial with your custom data, check out how to [Create an image dataset](https://huggingface.co/docs/datasets/image_dataset#loading-script)
guide in the 🤗 Datasets documentation.


key: _node_content
string_value:
{"id_": "c72d755f-1c90-4e31-a34e-c3beaf6043ff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1610bb4d-e366-4424-af9c-43c772a8ca45", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "a80e884ceead7f0e3dcced861a919271149261d9fd9cf34b85ce2c801d511af7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e1fbac6-e51e-4dd0-85a6-044c5470e6bd", "node_type": "1", "metadata": {}, "hash": "0acb8adae8bd87e2910c4ca9df30d516a69aad0f0ce8d91dd6dad342761e8321", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1375, "end_char_idx": 3477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model can be used for several downstream tasks. For the VQA task, a classifier 
head is placed on top (a linear layer on top of the final hidden state of the `[CLS]` token) and randomly initialized. 
Visual Question Answering is thus treated as a **classification problem**.

More recent models, such as BLIP, BLIP-2, and InstructBLIP, treat VQA as a generative task. Later in this guide we 
illustrate how to use them for zero-shot VQA inference. 

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q transformers datasets
```

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define the model checkpoint as a global variable.

```py
>>> model_checkpoint = "dandelin/vilt-b32-mlm"
```

## Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering `Graphcore/vqa` dataset. 
You can find the full dataset on [🤗 Hub](https://huggingface.co/datasets/Graphcore/vqa).

As an alternative to the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa), you can download the 
same data manually from the official [VQA dataset page](https://visualqa.org/download.html). If you prefer to follow the 
tutorial with your custom data, check out how to [Create an image dataset](https://huggingface.co/docs/datasets/image_dataset#loading-script)
guide in the 🤗 Datasets documentation.  

Let's load the first 200 examples from the validation split and explore the dataset's features:  

```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("Graphcore/vqa", split="validation[:200]")
>>> dataset
Dataset({
    features: ['question', 'question_type', 'question_id', 'image_id', 'answer_type', 'label'],
    num_rows: 200
})
```

Let's take a look at an example to understand the dataset's features:

```py
>>> dataset[0]
{'question': 'Where is he looking?


key: _node_content
string_value:
{"id_": "8e1fbac6-e51e-4dd0-85a6-044c5470e6bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c72d755f-1c90-4e31-a34e-c3beaf6043ff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "61084e6f9dcff6c79e5b509c266c208e8c6e6a98fd766278db4ee44702c36042", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eaac7005-3633-4fb5-8d50-700e2e87ca5b", "node_type": "1", "metadata": {}, "hash": "a60eaf7042a93c1f1b08de60c13fafbe9cdcf01194db13ac0cc8e849cc5f4068", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1830, "end_char_idx": 4618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q transformers datasets
```

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define the model checkpoint as a global variable.

```py
>>> model_checkpoint = "dandelin/vilt-b32-mlm"
```

## Load the data

For illustration purposes, in this guide we use a very small sample of the annotated visual question answering `Graphcore/vqa` dataset. 
You can find the full dataset on [🤗 Hub](https://huggingface.co/datasets/Graphcore/vqa).

As an alternative to the [`Graphcore/vqa` dataset](https://huggingface.co/datasets/Graphcore/vqa), you can download the 
same data manually from the official [VQA dataset page](https://visualqa.org/download.html). If you prefer to follow the 
tutorial with your custom data, check out how to [Create an image dataset](https://huggingface.co/docs/datasets/image_dataset#loading-script)
guide in the 🤗 Datasets documentation.  

Let's load the first 200 examples from the validation split and explore the dataset's features:  

```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("Graphcore/vqa", split="validation[:200]")
>>> dataset
Dataset({
    features: ['question', 'question_type', 'question_id', 'image_id', 'answer_type', 'label'],
    num_rows: 200
})
```

Let's take a look at an example to understand the dataset's features:

```py
>>> dataset[0]
{'question': 'Where is he looking?',
 'question_type': 'none of the above',
 'question_id': 262148000,
 'image_id': '/root/.cache/huggingface/datasets/downloads/extracted/ca733e0e000fb2d7a09fbcc94dbfe7b5a30750681d0e965f8e0a23b1c2f98c75/val2014/COCO_val2014_000000262148.jpg',
 'answer_type': 'other',
 'label': {'ids': ['at table', 'down', 'skateboard', 'table'],
  'weights': [0.30000001192092896,
   1.0,
   0.30000001192092896,
   0.30000001192092896]}}
```

The features relevant to the task include: 
* `question`: the question to be answered from the image
* `image_id`: the path to the image the question refers to
* `label`: the annotations

We can remove the rest of the features as they won't be necessary: 

```py 
>>> dataset = dataset.remove_columns(['question_type', 'question_id', 'answer_type'])
```

As you can see, the `label` feature contains several answers to the same question (called `ids` here) collected by different human annotators. 
This is because the answer to a question can be subjective. In this case, the question is "where is he looking?". Some people 
annotated this with "down", others with "at table", another one with "skateboard", etc.


key: _node_content
string_value:
{"id_": "eaac7005-3633-4fb5-8d50-700e2e87ca5b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e1fbac6-e51e-4dd0-85a6-044c5470e6bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "1ce75d49988fd46d5f6e0a98e1f67159320ddacd47ef77731ad63c9341773c39", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45d8942b-6d8d-4a13-b9ec-662ed33d9002", "node_type": "1", "metadata": {}, "hash": "847bf8da481c4cf10b0f7305232170266be8f2595332e18eab5160d0faaf62e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3477, "end_char_idx": 6306, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
',
 'question_type': 'none of the above',
 'question_id': 262148000,
 'image_id': '/root/.cache/huggingface/datasets/downloads/extracted/ca733e0e000fb2d7a09fbcc94dbfe7b5a30750681d0e965f8e0a23b1c2f98c75/val2014/COCO_val2014_000000262148.jpg',
 'answer_type': 'other',
 'label': {'ids': ['at table', 'down', 'skateboard', 'table'],
  'weights': [0.30000001192092896,
   1.0,
   0.30000001192092896,
   0.30000001192092896]}}
```

The features relevant to the task include: 
* `question`: the question to be answered from the image
* `image_id`: the path to the image the question refers to
* `label`: the annotations

We can remove the rest of the features as they won't be necessary: 

```py 
>>> dataset = dataset.remove_columns(['question_type', 'question_id', 'answer_type'])
```

As you can see, the `label` feature contains several answers to the same question (called `ids` here) collected by different human annotators. 
This is because the answer to a question can be subjective. In this case, the question is "where is he looking?". Some people 
annotated this with "down", others with "at table", another one with "skateboard", etc. 

Take a look at the image and consider which answer would you give:

```python
>>> from PIL import Image

>>> image = Image.open(dataset[0]['image_id'])
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/vqa-example.png" alt="VQA Image Example"/>
</div>

Due to the questions' and answers' ambiguity, datasets like this are treated as a multi-label classification problem (as 
multiple answers are possibly valid). Moreover, rather than just creating a one-hot encoded vector, one creates a 
soft encoding, based on the number of times a certain answer appeared in the annotations.

For instance, in the example above, because the answer "down" is selected way more often than other answers, it has a 
score (called `weight` in the dataset) of 1.0, and the rest of the answers have scores < 1.0. 

To later instantiate the model with an appropriate classification head, let's create two dictionaries: one that maps 
the label name to an integer and vice versa:

```py
>>> import itertools

>>> labels = [item['ids'] for item in dataset['label']]
>>> flattened_labels = list(itertools.chain(*labels))
>>> unique_labels = list(set(flattened_labels))

>>> label2id = {label: idx for idx, label in enumerate(unique_labels)}
>>> id2label = {idx: label for label, idx in label2id.items()} 
```

Now that we have the mappings, we can replace the string answers with their ids, and flatten the dataset for a more convenient further preprocessing. 

```python
>>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs


key: _node_content
string_value:
{"id_": "45d8942b-6d8d-4a13-b9ec-662ed33d9002", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eaac7005-3633-4fb5-8d50-700e2e87ca5b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "96dba756c088f18794f8065dd1ef540c842f11c37fd2f4adc18cbbb8d87fccba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "580dadb0-cbb6-4a2d-8658-71c315c90784", "node_type": "1", "metadata": {}, "hash": "dd80bb7eceaf85158ad0f552cdf468a66c929f97b1b4bc05533888e3bd28b088", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4621, "end_char_idx": 7755, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Take a look at the image and consider which answer would you give:

```python
>>> from PIL import Image

>>> image = Image.open(dataset[0]['image_id'])
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/vqa-example.png" alt="VQA Image Example"/>
</div>

Due to the questions' and answers' ambiguity, datasets like this are treated as a multi-label classification problem (as 
multiple answers are possibly valid). Moreover, rather than just creating a one-hot encoded vector, one creates a 
soft encoding, based on the number of times a certain answer appeared in the annotations.

For instance, in the example above, because the answer "down" is selected way more often than other answers, it has a 
score (called `weight` in the dataset) of 1.0, and the rest of the answers have scores < 1.0. 

To later instantiate the model with an appropriate classification head, let's create two dictionaries: one that maps 
the label name to an integer and vice versa:

```py
>>> import itertools

>>> labels = [item['ids'] for item in dataset['label']]
>>> flattened_labels = list(itertools.chain(*labels))
>>> unique_labels = list(set(flattened_labels))

>>> label2id = {label: idx for idx, label in enumerate(unique_labels)}
>>> id2label = {idx: label for label, idx in label2id.items()} 
```

Now that we have the mappings, we can replace the string answers with their ids, and flatten the dataset for a more convenient further preprocessing. 

```python
>>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs


>>> dataset = dataset.map(replace_ids)
>>> flat_dataset = dataset.flatten()
>>> flat_dataset.features
{'question': Value(dtype='string', id=None),
 'image_id': Value(dtype='string', id=None),
 'label.ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),
 'label.weights': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None)}
```

## Preprocessing data

The next step is to load a ViLT processor to prepare the image and text data for the model. 
[`ViltProcessor`] wraps a BERT tokenizer and ViLT image processor into a convenient single processor:

```py 
>>> from transformers import ViltProcessor

>>> processor = ViltProcessor.from_pretrained(model_checkpoint)
```

To preprocess the data we need to encode the images and questions using the [`ViltProcessor`]. The processor will use 
the [`BertTokenizerFast`] to tokenize the text and create `input_ids`, `attention_mask` and `token_type_ids` for the text data. 
As for images, the processor will leverage [`ViltImageProcessor`] to resize and normalize the image, and create `pixel_values` and `pixel_mask`.

All these preprocessing steps are done under the hood, we only need to call the `processor`. However, we still need to 
prepare the target labels. In this representation, each element corresponds to a possible answer (label). For correct answers, the element holds 
their respective score (weight), while the remaining elements are set to zero.


key: _node_content
string_value:
{"id_": "580dadb0-cbb6-4a2d-8658-71c315c90784", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45d8942b-6d8d-4a13-b9ec-662ed33d9002", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "6a4d9a8d6075c13f8dea0fbc27d5e86a4e135973ddbff061d72d04ac4d26725c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c36ee6e-5ceb-4292-ac7a-17aab8a7f1c7", "node_type": "1", "metadata": {}, "hash": "f8255841ffdfce85f3c0721f1214c436bf83f0228fa8a8c305268984bab2b930", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6171, "end_char_idx": 9561, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> def replace_ids(inputs):
...   inputs["label"]["ids"] = [label2id[x] for x in inputs["label"]["ids"]]
...   return inputs


>>> dataset = dataset.map(replace_ids)
>>> flat_dataset = dataset.flatten()
>>> flat_dataset.features
{'question': Value(dtype='string', id=None),
 'image_id': Value(dtype='string', id=None),
 'label.ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),
 'label.weights': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None)}
```

## Preprocessing data

The next step is to load a ViLT processor to prepare the image and text data for the model. 
[`ViltProcessor`] wraps a BERT tokenizer and ViLT image processor into a convenient single processor:

```py 
>>> from transformers import ViltProcessor

>>> processor = ViltProcessor.from_pretrained(model_checkpoint)
```

To preprocess the data we need to encode the images and questions using the [`ViltProcessor`]. The processor will use 
the [`BertTokenizerFast`] to tokenize the text and create `input_ids`, `attention_mask` and `token_type_ids` for the text data. 
As for images, the processor will leverage [`ViltImageProcessor`] to resize and normalize the image, and create `pixel_values` and `pixel_mask`.

All these preprocessing steps are done under the hood, we only need to call the `processor`. However, we still need to 
prepare the target labels. In this representation, each element corresponds to a possible answer (label). For correct answers, the element holds 
their respective score (weight), while the remaining elements are set to zero.

The following function applies the `processor` to the images and questions and formats the labels as described above:

```py
>>> import torch

>>> def preprocess_data(examples):
...     image_paths = examples['image_id']
...     images = [Image.open(image_path) for image_path in image_paths]
...     texts = examples['question']    

...     encoding = processor(images, texts, padding="max_length", truncation=True, return_tensors="pt")

...     for k, v in encoding.items():
...           encoding[k] = v.squeeze()
    
...     targets = []

...     for labels, scores in zip(examples['label.ids'], examples['label.weights']):
...         target = torch.zeros(len(id2label))

...         for label, score in zip(labels, scores):
...             target[label] = score
      
...         targets.append(target)

...     encoding["labels"] = targets
    
...     return encoding
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.map`] function. You can speed up `map` by 
setting `batched=True` to process multiple elements of the dataset at once. At this point, feel free to remove the columns you don't need.

```py
>>> processed_dataset = flat_dataset.map(preprocess_data, batched=True, remove_columns=['question','question_type',  'question_id', 'image_id', 'answer_type', 'label.ids', 'label.weights'])
>>> processed_dataset
Dataset({
    features: ['input_ids', 'token_type_ids', 'attention_mask', 'pixel_values', 'pixel_mask', 'labels'],
    num_rows: 200
})
```

As a final step, create a batch of examples using [`DefaultDataCollator`]:

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

## Train the model

You’re ready to start training your model now! Load ViLT with [`ViltForQuestionAnswering`].


key: _node_content
string_value:
{"id_": "7c36ee6e-5ceb-4292-ac7a-17aab8a7f1c7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "580dadb0-cbb6-4a2d-8658-71c315c90784", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "3910178bf9ea01dc3e7ba7bd5f6efc34cc493f3a43ede3b1076bd1de19165df2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ea42ff4-39be-4bb8-b18f-31fd0305f349", "node_type": "1", "metadata": {}, "hash": "d834cd5660a1e4fb5399a7713ea8000b62145920c168ee083a0488363d97b325", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8745, "end_char_idx": 11830, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can speed up `map` by 
setting `batched=True` to process multiple elements of the dataset at once. At this point, feel free to remove the columns you don't need.

```py
>>> processed_dataset = flat_dataset.map(preprocess_data, batched=True, remove_columns=['question','question_type',  'question_id', 'image_id', 'answer_type', 'label.ids', 'label.weights'])
>>> processed_dataset
Dataset({
    features: ['input_ids', 'token_type_ids', 'attention_mask', 'pixel_values', 'pixel_mask', 'labels'],
    num_rows: 200
})
```

As a final step, create a batch of examples using [`DefaultDataCollator`]:

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

## Train the model

You’re ready to start training your model now! Load ViLT with [`ViltForQuestionAnswering`]. Specify the number of labels 
along with the label mappings:

```py
>>> from transformers import ViltForQuestionAnswering

>>> model = ViltForQuestionAnswering.from_pretrained(model_checkpoint, num_labels=len(id2label), id2label=id2label, label2id=label2id)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]:

```py
>>> from transformers import TrainingArguments

>>> repo_id = "MariaK/vilt_finetuned_200"

>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...     logging_steps=50,
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
```

2. Pass the training arguments to [`Trainer`] along with the model, dataset, processor, and data collator.

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=processed_dataset,
...     tokenizer=processor,
... )
```

3. Call [`~Trainer.train`] to finetune your model.

```py
>>> trainer.train() 
```

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method to share your final model on the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

Now that you have fine-tuned a ViLT model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your fine-tuned model for inference is to use it in a [`Pipeline`].

```py
>>> from transformers import pipeline

>>> pipe = pipeline("visual-question-answering", model="MariaK/vilt_finetuned_200")
```

The model in this guide has only been trained on 200 examples, so don't expect a lot from it. Let's see if it at least 
learned something from the data and take the first example from the dataset to illustrate inference:

```py
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
>>> print(question)
>>> pipe(image, question, top_k=1)
"Where is he looking?"
[{'score': 0.5498199462890625, 'answer': 'down'}]
```

Even though not very confident, the model indeed has learned something.


key: _node_content
string_value:
{"id_": "7ea42ff4-39be-4bb8-b18f-31fd0305f349", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c36ee6e-5ceb-4292-ac7a-17aab8a7f1c7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "3f6d457b0083514f057206c025ca5933864f27f9eebd2c298a3f9a87a23dd995", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aba742c9-4691-42fb-b9ad-e947c21bbc10", "node_type": "1", "metadata": {}, "hash": "e674133eefc00b096425314c7c1f55afe77986fa70e3d17259aa0990bd9b8935", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10348, "end_char_idx": 13430, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass the training arguments to [`Trainer`] along with the model, dataset, processor, and data collator.

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=processed_dataset,
...     tokenizer=processor,
... )
```

3. Call [`~Trainer.train`] to finetune your model.

```py
>>> trainer.train() 
```

Once training is completed, share your model to the Hub with the [`~Trainer.push_to_hub`] method to share your final model on the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

Now that you have fine-tuned a ViLT model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your fine-tuned model for inference is to use it in a [`Pipeline`].

```py
>>> from transformers import pipeline

>>> pipe = pipeline("visual-question-answering", model="MariaK/vilt_finetuned_200")
```

The model in this guide has only been trained on 200 examples, so don't expect a lot from it. Let's see if it at least 
learned something from the data and take the first example from the dataset to illustrate inference:

```py
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
>>> print(question)
>>> pipe(image, question, top_k=1)
"Where is he looking?"
[{'score': 0.5498199462890625, 'answer': 'down'}]
```

Even though not very confident, the model indeed has learned something. With more examples and longer training, you'll get far better results!

You can also manually replicate the results of the pipeline if you'd like:
1. Take an image and a question, prepare them for the model using the processor from your model.
2. Forward the result or preprocessing through the model.
3. From the logits, get the most likely answer's id, and find the actual answer in the `id2label`.

```py
>>> processor = ViltProcessor.from_pretrained("MariaK/vilt_finetuned_200")

>>> image = Image.open(example['image_id'])
>>> question = example['question']

>>> # prepare inputs
>>> inputs = processor(image, question, return_tensors="pt")

>>> model = ViltForQuestionAnswering.from_pretrained("MariaK/vilt_finetuned_200")

>>> # forward pass
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits = outputs.logits
>>> idx = logits.argmax(-1).item()
>>> print("Predicted answer:", model.config.id2label[idx])
Predicted answer: down
```

## Zero-shot VQA

The previous model treated VQA as a classification task. Some recent models, such as BLIP, BLIP-2, and InstructBLIP approach 
VQA as a generative task. Let's take [BLIP-2](../model_doc/blip-2) as an example. It introduced a new visual-language pre-training 
paradigm in which any combination of pre-trained vision encoder and LLM can be used (learn more in the [BLIP-2 blog post](https://huggingface.co/blog/blip-2)). 
This enables achieving state-of-the-art results on multiple visual-language tasks including visual question answering. 

Let's illustrate how you can use this model for VQA. First, let's load the model.


key: _node_content
string_value:
{"id_": "aba742c9-4691-42fb-b9ad-e947c21bbc10", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ea42ff4-39be-4bb8-b18f-31fd0305f349", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "bb78642c4fe7af17fbb96bbf466994e17e83ff95d5a5b43381a100dffed3716f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2510247f-1d57-426d-9194-ec7246190276", "node_type": "1", "metadata": {}, "hash": "be8b95418d0a422bce27dba3cbf966c13cc716a308af99ffa88e44b601d5f4d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11831, "end_char_idx": 15013, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With more examples and longer training, you'll get far better results!

You can also manually replicate the results of the pipeline if you'd like:
1. Take an image and a question, prepare them for the model using the processor from your model.
2. Forward the result or preprocessing through the model.
3. From the logits, get the most likely answer's id, and find the actual answer in the `id2label`.

```py
>>> processor = ViltProcessor.from_pretrained("MariaK/vilt_finetuned_200")

>>> image = Image.open(example['image_id'])
>>> question = example['question']

>>> # prepare inputs
>>> inputs = processor(image, question, return_tensors="pt")

>>> model = ViltForQuestionAnswering.from_pretrained("MariaK/vilt_finetuned_200")

>>> # forward pass
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits = outputs.logits
>>> idx = logits.argmax(-1).item()
>>> print("Predicted answer:", model.config.id2label[idx])
Predicted answer: down
```

## Zero-shot VQA

The previous model treated VQA as a classification task. Some recent models, such as BLIP, BLIP-2, and InstructBLIP approach 
VQA as a generative task. Let's take [BLIP-2](../model_doc/blip-2) as an example. It introduced a new visual-language pre-training 
paradigm in which any combination of pre-trained vision encoder and LLM can be used (learn more in the [BLIP-2 blog post](https://huggingface.co/blog/blip-2)). 
This enables achieving state-of-the-art results on multiple visual-language tasks including visual question answering. 

Let's illustrate how you can use this model for VQA. First, let's load the model. Here we'll explicitly send the model to a 
GPU, if available, which we didn't need to do earlier when training, as [`Trainer`] handles this automatically: 

```py
>>> from transformers import AutoProcessor, Blip2ForConditionalGeneration
>>> import torch

>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
```

The model takes image and text as input, so let's use the exact same image/question pair from the first example in the VQA dataset: 

```py 
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
```

To use BLIP-2 for visual question answering task, the textual prompt has to follow a specific format: `Question: {} Answer:`.

```py
>>> prompt = f"Question: {question} Answer:" 
```

Now we need to preprocess the image/prompt with the model's processor, pass the processed input through the model, and decode the output:

```py
>>> inputs = processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)

>>> generated_ids = model.generate(**inputs, max_new_tokens=10)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
"He is looking at the crowd" 
```

As you can see, the model recognized the crowd, and the direction of the face (looking down), however, it seems to miss 
the fact the crowd is behind the skater.


key: _node_content
string_value:
{"id_": "2510247f-1d57-426d-9194-ec7246190276", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2f4e4bf6-20dc-41cd-8c45-0e298efabe77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "4beeead43ec2efaea2518803588709d06bb6b4bdfffceb248f2173f3e682aa8c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aba742c9-4691-42fb-b9ad-e947c21bbc10", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/visual_question_answering", "title": "Visual Question Answering", "tokens": 3788, "retrieve_doc": true, "source": "transformers"}, "hash": "ffdd20022e74a54c57bca75163ab1dcebc1af81d01c226709daaf99cb280ab7f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13402, "end_char_idx": 15138, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
First, let's load the model. Here we'll explicitly send the model to a 
GPU, if available, which we didn't need to do earlier when training, as [`Trainer`] handles this automatically: 

```py
>>> from transformers import AutoProcessor, Blip2ForConditionalGeneration
>>> import torch

>>> processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
>>> model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16)
>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)
```

The model takes image and text as input, so let's use the exact same image/question pair from the first example in the VQA dataset: 

```py 
>>> example = dataset[0]
>>> image = Image.open(example['image_id'])
>>> question = example['question']
```

To use BLIP-2 for visual question answering task, the textual prompt has to follow a specific format: `Question: {} Answer:`.

```py
>>> prompt = f"Question: {question} Answer:" 
```

Now we need to preprocess the image/prompt with the model's processor, pass the processed input through the model, and decode the output:

```py
>>> inputs = processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)

>>> generated_ids = model.generate(**inputs, max_new_tokens=10)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
>>> print(generated_text)
"He is looking at the crowd" 
```

As you can see, the model recognized the crowd, and the direction of the face (looking down), however, it seems to miss 
the fact the crowd is behind the skater. Still, in cases where acquiring human-annotated datasets is not feasible, this 
approach can quickly produce useful results.


key: _node_content
string_value:
{"id_": "53078042-266d-47b5-8cd9-c4dbb723b32d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f39cbd72-6b3d-4e26-8904-f16ed89a6b68", "node_type": "1", "metadata": {}, "hash": "f07cf64d6a3f473b4ed9581a79a687b78d7edcf3fe101b81cfc7d3baac827e1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3039, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image classification

[[open-in-colab]]

<Youtube id="tjAIM7BOYhw"/>

Image classification assigns a label or class to an image. Unlike text or audio classification, the inputs are the
pixel values that comprise an image. There are many applications for image classification, such as detecting damage
after a natural disaster, monitoring crop health, or helping screen medical images for signs of disease.

This guide illustrates how to:

1. Fine-tune [ViT](model_doc/vit) on the [Food-101](https://huggingface.co/datasets/food101) dataset to classify a food item in an image.
2. Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-classification)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate accelerate pillow torchvision scikit-learn
```

We encourage you to log in to your Hugging Face account to upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load Food-101 dataset

Start by loading a smaller subset of the Food-101 dataset from the 🤗 Datasets library. This will give you a chance to
experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> food = load_dataset("food101", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> food = food.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> food["train"][0]
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x512 at 0x7F52AFC8AC50>,
 'label': 79}
```

Each example in the dataset has two fields:

- `image`: a PIL image of the food item
- `label`: the label class of the food item

To make it easier for the model to get the label name from the label id, create a dictionary that maps the label name
to an integer and vice versa:

```py
>>> labels = food["train"].features["label"].names
>>> label2id, id2label = dict(), dict()
>>> for i, label in enumerate(labels):
...     label2id[label] = str(i)
...     id2label[str(i)] = label
```

Now you can convert the label id to a label name:

```py
>>> id2label[str(79)]
'prime_rib'
```

## Preprocess

The next step is to load a ViT image processor to process the image into a tensor:

```py
>>> from transformers import AutoImageProcessor

>>> checkpoint = "google/vit-base-patch16-224-in21k"
>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
```

<frameworkcontent>
<pt>
Apply some image transformations to the images to make the model more robust against overfitting. Here you'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module, but you can also use any image library you like.


key: _node_content
string_value:
{"id_": "f39cbd72-6b3d-4e26-8904-f16ed89a6b68", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "53078042-266d-47b5-8cd9-c4dbb723b32d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "886a0267b29d0b07c34d25e05b8c11e9e861874cd84d4d1995ba39948bbdda5f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00ba3b6b-850f-4121-8e93-97f9fc3bb900", "node_type": "1", "metadata": {}, "hash": "f08f3d0f622ac08c0a2924aab247c9ae7ab6c449789efc2936b1da623800c471", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2887, "end_char_idx": 6194, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here you'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module, but you can also use any image library you like.

Crop a random part of the image, resize it, and normalize it with the image mean and standard deviation:

```py
>>> from torchvision.transforms import RandomResizedCrop, Compose, Normalize, ToTensor

>>> normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )
>>> _transforms = Compose([RandomResizedCrop(size), ToTensor(), normalize])
```

Then create a preprocessing function to apply the transforms and return the `pixel_values` - the inputs to the model - of the image:

```py
>>> def transforms(examples):
...     examples["pixel_values"] = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     del examples["image"]
...     return examples
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.with_transform`] method. The transforms are applied on the fly when you load an element of the dataset:

```py
>>> food = food.with_transform(transforms)
```

Now create a batch of examples using [`DefaultDataCollator`]. Unlike other data collators in 🤗 Transformers, the `DefaultDataCollator` does not apply additional preprocessing such as padding.

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```
</pt>
</frameworkcontent>


<frameworkcontent>
<tf>

To avoid overfitting and to make the model more robust, add some data augmentation to the training part of the dataset.
Here we use Keras preprocessing layers to define the transformations for the training data (includes data augmentation),
and transformations for the validation data (only center cropping, resizing and normalizing). You can use `tf.image`or
any other library you prefer.

```py
>>> from tensorflow import keras
>>> from tensorflow.keras import layers

>>> size = (image_processor.size["height"], image_processor.size["width"])

>>> train_data_augmentation = keras.Sequential(
...     [
...         layers.RandomCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...         layers.RandomFlip("horizontal"),
...         layers.RandomRotation(factor=0.02),
...         layers.RandomZoom(height_factor=0.2, width_factor=0.2),
...     ],
...     name="train_data_augmentation",
... )

>>> val_data_augmentation = keras.Sequential(
...     [
...         layers.CenterCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...     ],
...     name="val_data_augmentation",
... )
```

Next, create functions to apply appropriate transformations to a batch of images, instead of one image at a time.

```py
>>> import numpy as np
>>> import tensorflow as tf
>>> from PIL import Image


>>> def convert_to_tf_tensor(image: Image):
...     np_image = np.array(image)
...     tf_image = tf.convert_to_tensor(np_image)
...     # `expand_dims()` is used to add a batch dimension since
...     # the TF augmentation layers operates on batched inputs.
...     return tf.expand_dims(tf_image, 0)


key: _node_content
string_value:
{"id_": "00ba3b6b-850f-4121-8e93-97f9fc3bb900", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f39cbd72-6b3d-4e26-8904-f16ed89a6b68", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "3649120c3ec7ba7617dcfdf03893c95ee811b2fae54ac780dfc2dbc0c445a08c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d2b0086-bd56-4f15-80dc-ca2a183bfc0b", "node_type": "1", "metadata": {}, "hash": "856f897edd6dc468ad9d0b428d3806849118fbed4b8e33cd1a5467c5ab711f96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4510, "end_char_idx": 6558, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<tf>

To avoid overfitting and to make the model more robust, add some data augmentation to the training part of the dataset.
Here we use Keras preprocessing layers to define the transformations for the training data (includes data augmentation),
and transformations for the validation data (only center cropping, resizing and normalizing). You can use `tf.image`or
any other library you prefer.

```py
>>> from tensorflow import keras
>>> from tensorflow.keras import layers

>>> size = (image_processor.size["height"], image_processor.size["width"])

>>> train_data_augmentation = keras.Sequential(
...     [
...         layers.RandomCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...         layers.RandomFlip("horizontal"),
...         layers.RandomRotation(factor=0.02),
...         layers.RandomZoom(height_factor=0.2, width_factor=0.2),
...     ],
...     name="train_data_augmentation",
... )

>>> val_data_augmentation = keras.Sequential(
...     [
...         layers.CenterCrop(size[0], size[1]),
...         layers.Rescaling(scale=1.0 / 127.5, offset=-1),
...     ],
...     name="val_data_augmentation",
... )
```

Next, create functions to apply appropriate transformations to a batch of images, instead of one image at a time.

```py
>>> import numpy as np
>>> import tensorflow as tf
>>> from PIL import Image


>>> def convert_to_tf_tensor(image: Image):
...     np_image = np.array(image)
...     tf_image = tf.convert_to_tensor(np_image)
...     # `expand_dims()` is used to add a batch dimension since
...     # the TF augmentation layers operates on batched inputs.
...     return tf.expand_dims(tf_image, 0)


>>> def preprocess_train(example_batch):
...     """Apply train_transforms across a batch."""
...     images = [
...         train_data_augmentation(convert_to_tf_tensor(image.convert("RGB"))) for image in example_batch["image"]
...     ]
...     example_batch["pixel_values"] = [tf.transpose(tf.squeeze(image)) for image in images]
...     return example_batch


key: _node_content
string_value:
{"id_": "6d2b0086-bd56-4f15-80dc-ca2a183bfc0b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00ba3b6b-850f-4121-8e93-97f9fc3bb900", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "b6d38bd8fb5da8e0329116285c21c72c9bff5e9e560a664d358e106f2aa256a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ca0e485-0da0-4095-b7c1-75321e3cad0b", "node_type": "1", "metadata": {}, "hash": "8ebf729f7c3e3d478a005928d544cdf05d8f33e29259247aa51839f57548fbca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5893, "end_char_idx": 9273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def convert_to_tf_tensor(image: Image):
...     np_image = np.array(image)
...     tf_image = tf.convert_to_tensor(np_image)
...     # `expand_dims()` is used to add a batch dimension since
...     # the TF augmentation layers operates on batched inputs.
...     return tf.expand_dims(tf_image, 0)


>>> def preprocess_train(example_batch):
...     """Apply train_transforms across a batch."""
...     images = [
...         train_data_augmentation(convert_to_tf_tensor(image.convert("RGB"))) for image in example_batch["image"]
...     ]
...     example_batch["pixel_values"] = [tf.transpose(tf.squeeze(image)) for image in images]
...     return example_batch


... def preprocess_val(example_batch):
...     """Apply val_transforms across a batch."""
...     images = [
...         val_data_augmentation(convert_to_tf_tensor(image.convert("RGB"))) for image in example_batch["image"]
...     ]
...     example_batch["pixel_values"] = [tf.transpose(tf.squeeze(image)) for image in images]
...     return example_batch
```

Use 🤗 Datasets [`~datasets.Dataset.set_transform`] to apply the transformations on the fly:

```py
food["train"].set_transform(preprocess_train)
food["test"].set_transform(preprocess_val)
```

As a final preprocessing step, create a batch of examples using `DefaultDataCollator`. Unlike other data collators in 🤗 Transformers, the
`DefaultDataCollator` does not apply additional preprocessing, such as padding.

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator(return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load an
evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load
the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you set up your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load ViT with [`AutoModelForImageClassification`]. Specify the number of labels along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForImageClassification, TrainingArguments, Trainer

>>> model = AutoModelForImageClassification.from_pretrained(
...     checkpoint,
...     num_labels=len(labels),
...     id2label=id2label,
...     label2id=label2id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "9ca0e485-0da0-4095-b7c1-75321e3cad0b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d2b0086-bd56-4f15-80dc-ca2a183bfc0b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "c5768e828cc7050fb90d8c343bcff898768691e388c4078a72972134fd9a589f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e52b6025-b33c-4739-a28e-1ce83e5bd7cf", "node_type": "1", "metadata": {}, "hash": "aad4b2649d3d96efc2568844bc5596a332a52834936a2874b294480a27ab0e45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8180, "end_char_idx": 11434, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you set up your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load ViT with [`AutoModelForImageClassification`]. Specify the number of labels along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForImageClassification, TrainingArguments, Trainer

>>> model = AutoModelForImageClassification.from_pretrained(
...     checkpoint,
...     num_labels=len(labels),
...     id2label=id2label,
...     label2id=label2id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. It is important you don't remove unused columns because that'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_food_model",
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=food["train"],
...     eval_dataset=food["test"],
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>

<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pre-trained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.


key: _node_content
string_value:
{"id_": "e52b6025-b33c-4739-a28e-1ce83e5bd7cf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ca0e485-0da0-4095-b7c1-75321e3cad0b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "6c78d8e5e5f01fe72aaacb24efb4f27b6ca6ec67f83f902d495fcce52b421b17", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2e02770-bb8d-4ff4-9506-adc0fef8167a", "node_type": "1", "metadata": {}, "hash": "04349a153348402a3382c78e9f4a8686e0ab9bf73f30b160645e7f4168f12582", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9943, "end_char_idx": 11581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_food_model",
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=food["train"],
...     eval_dataset=food["test"],
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>

<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pre-trained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks and use the `fit()` method to run the training.
6. Upload your model to 🤗 Hub to share with the community.


key: _node_content
string_value:
{"id_": "d2e02770-bb8d-4ff4-9506-adc0fef8167a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e52b6025-b33c-4739-a28e-1ce83e5bd7cf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "266e90ba92f789967ce704cecc865da9cbe4b6384dce97945358cf3e5dd774ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e452998-66bc-4937-b2e2-811f650e2438", "node_type": "1", "metadata": {}, "hash": "91f91fabf466cad9594a8578f516903c3aa88d8bb82fac7060a63121cbab67c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9995, "end_char_idx": 13354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_food_model",
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=food["train"],
...     eval_dataset=food["test"],
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>

<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pre-trained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks and use the `fit()` method to run the training.
6. Upload your model to 🤗 Hub to share with the community.

Start by defining the hyperparameters, optimizer and learning rate schedule:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_epochs = 5
>>> num_train_steps = len(food["train"]) * num_epochs
>>> learning_rate = 3e-5
>>> weight_decay_rate = 0.01

>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... )
```

Then, load ViT with [`TFAutoModelForImageClassification`] along with the label mappings:

```py
>>> from transformers import TFAutoModelForImageClassification

>>> model = TFAutoModelForImageClassification.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
... )
```

Convert your datasets to the `tf.data.Dataset` format using the [`~datasets.Dataset.to_tf_dataset`] and your `data_collator`:

```py
>>> # converting our train dataset to tf.data.Dataset
>>> tf_train_dataset = food["train"].to_tf_dataset(
...     columns="pixel_values", label_cols="label", shuffle=True, batch_size=batch_size, collate_fn=data_collator
... )

>>> # converting our test dataset to tf.data.Dataset
>>> tf_eval_dataset = food["test"].to_tf_dataset(
...     columns="pixel_values", label_cols="label", shuffle=True, batch_size=batch_size, collate_fn=data_collator
... )
```

Configure the model for training with `compile()`:

```py
>>> from tensorflow.keras.losses import SparseCategoricalCrossentropy

>>> loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
>>> model.compile(optimizer=optimizer, loss=loss)
```

To compute the accuracy from the predictions and push your model to the 🤗 Hub, use [Keras callbacks](../main_classes/keras_callbacks).


key: _node_content
string_value:
{"id_": "5e452998-66bc-4937-b2e2-811f650e2438", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2e02770-bb8d-4ff4-9506-adc0fef8167a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "bb7d16ec35009b91499c801a449ace96344d033087e41cb08ed7d5d17a0c4676", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d8dd4c7-100e-47d2-a2ae-2ed5b4df1b99", "node_type": "1", "metadata": {}, "hash": "79a30dbd43256465653a5ee56e274196f9a0b4a59e3c0a6aacc274ab0e8c9ae5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13355, "end_char_idx": 15022, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass your `compute_metrics` function to [KerasMetricCallback](../main_classes/keras_callbacks#transformers.KerasMetricCallback),
and use the [PushToHubCallback](../main_classes/keras_callbacks#transformers.PushToHubCallback) to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_eval_dataset)
>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="food_classifier",
...     tokenizer=image_processor,
...     save_strategy="no",
... )
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_eval_dataset, epochs=num_epochs, callbacks=callbacks)
Epoch 1/5
250/250 [==============================] - 313s 1s/step - loss: 2.5623 - val_loss: 1.4161 - accuracy: 0.9290
Epoch 2/5
250/250 [==============================] - 265s 1s/step - loss: 0.9181 - val_loss: 0.6808 - accuracy: 0.9690
Epoch 3/5
250/250 [==============================] - 252s 1s/step - loss: 0.3910 - val_loss: 0.4303 - accuracy: 0.9820
Epoch 4/5
250/250 [==============================] - 251s 1s/step - loss: 0.2028 - val_loss: 0.3191 - accuracy: 0.9900
Epoch 5/5
250/250 [==============================] - 238s 949ms/step - loss: 0.1232 - val_loss: 0.3259 - accuracy: 0.9890
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8d8dd4c7-100e-47d2-a2ae-2ed5b4df1b99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79d912a7-d5d0-4c78-9856-3a9220ead88a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "7d9276d485ce3f5c84ce243d6affdad336f403f9a173068ecddf362aef8287c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e452998-66bc-4937-b2e2-811f650e2438", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "transformers"}, "hash": "f841f082bca60a0b50f8914bee775cdc719f2554e0435ace2f8255f19ccb27ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14050, "end_char_idx": 18122, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_eval_dataset, epochs=num_epochs, callbacks=callbacks)
Epoch 1/5
250/250 [==============================] - 313s 1s/step - loss: 2.5623 - val_loss: 1.4161 - accuracy: 0.9290
Epoch 2/5
250/250 [==============================] - 265s 1s/step - loss: 0.9181 - val_loss: 0.6808 - accuracy: 0.9690
Epoch 3/5
250/250 [==============================] - 252s 1s/step - loss: 0.3910 - val_loss: 0.4303 - accuracy: 0.9820
Epoch 4/5
250/250 [==============================] - 251s 1s/step - loss: 0.2028 - val_loss: 0.3191 - accuracy: 0.9900
Epoch 5/5
250/250 [==============================] - 238s 949ms/step - loss: 0.1232 - val_loss: 0.3259 - accuracy: 0.9890
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>


<Tip>

For a more in-depth example of how to finetune a model for image classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

</Tip>

## Inference

Great, now that you've fine-tuned a model, you can use it for inference!

Load an image you'd like to run inference on:

```py
>>> ds = load_dataset("food101", split="validation[:10]")
>>> image = ds["image"][0]
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png" alt="image of beignets"/>
</div>

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for image classification with your model, and pass your image to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("image-classification", model="my_awesome_food_model")
>>> classifier(image)
[{'score': 0.31856709718704224, 'label': 'beignets'},
 {'score': 0.015232225880026817, 'label': 'bruschetta'},
 {'score': 0.01519392803311348, 'label': 'chicken_wings'},
 {'score': 0.013022331520915031, 'label': 'pork_chop'},
 {'score': 0.012728818692266941, 'label': 'prime_rib'}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Load an image processor to preprocess the image and return the `input` as PyTorch tensors:

```py
>>> from transformers import AutoImageProcessor
>>> import torch

>>> image_processor = AutoImageProcessor.from_pretrained("my_awesome_food_model")
>>> inputs = image_processor(image, return_tensors="pt")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import AutoModelForImageClassification

>>> model = AutoModelForImageClassification.from_pretrained("my_awesome_food_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the predicted label with the highest probability, and use the model's `id2label` mapping to convert it to a label:

```py
>>> predicted_label = logits.argmax(-1).item()
>>> model.config.id2label[predicted_label]
'beignets'
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
Load an image processor to preprocess the image and return the `input` as TensorFlow tensors:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/food_classifier")
>>> inputs = image_processor(image, return_tensors="tf")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import TFAutoModelForImageClassification

>>> model = TFAutoModelForImageClassification.from_pretrained("MariaK/food_classifier")
>>> logits = model(**inputs).logits
```

Get the predicted label with the highest probability, and use the model's `id2label` mapping to convert it to a label:

```py
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
>>> model.config.id2label[predicted_class_id]
'beignets'
```

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "4169dcdc-04f0-4578-970c-713a72eb2d90", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1adf76d2-39b8-4c8c-8de3-84d00e5bb189", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "44b1eabc695f079866d1f46e61cd6aea2a966b045812ccdec9c9b361190b36af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58fcf0d4-8ec3-463a-872f-a69cc1713c35", "node_type": "1", "metadata": {}, "hash": "f43c9d37b11f915c9ea0a8ff53076de7693fb192863f789e66f4c4b65ed225e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Monocular depth estimation

Monocular depth estimation is a computer vision task that involves predicting the depth information of a scene from a
single image. In other words, it is the process of estimating the distance of objects in a scene from
a single camera viewpoint.

Monocular depth estimation has various applications, including 3D reconstruction, augmented reality, autonomous driving,
and robotics. It is a challenging task as it requires the model to understand the complex relationships between objects
in the scene and the corresponding depth information, which can be affected by factors such as lighting conditions,
occlusion, and texture. 

There are two main depth estimation categories:

- **Absolute depth estimation**: This task variant aims to provide exact depth measurements from the camera. The term is used interchangeably with metric depth estimation, where depth is provided in precise measurements in meters or feet. Absolute depth estimation models output depth maps with numerical values that represent real-world distances.

- **Relative depth estimation**: Relative depth estimation aims to predict the depth order of objects or points in a scene without providing the precise measurements. These models output a depth map that indicates which parts of the scene are closer or farther relative to each other without the actual distances to A and B.

In this guide, we will see how to infer with [Depth Anything V2](https://huggingface.co/depth-anything/Depth-Anything-V2-Large), a state-of-the-art zero-shot relative depth estimation model, and [ZoeDepth](https://huggingface.co/docs/transformers/main/en/model_doc/zoedepth), an absolute depth estimation model.

<Tip>

Check the [Depth Estimation](https://huggingface.co/tasks/depth-estimation) task page to view all compatible architectures and checkpoints.

</Tip>

Before we begin, we need to install the latest version of Transformers:

```bash
pip install -q -U transformers
```

## Depth estimation pipeline

The simplest way to try out inference with a model supporting depth estimation is to use the corresponding [`pipeline`].
Instantiate a pipeline from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads):

```py
>>> from transformers import pipeline
>>> import torch

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> checkpoint = "depth-anything/Depth-Anything-V2-base-hf"
>>> pipe = pipeline("depth-estimation", model=checkpoint, device=device)
```

Next, choose an image to analyze:

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Photo of a bee"/>
</div>

Pass the image to the pipeline.

```py
>>> predictions = pipe(image)
```

The pipeline returns a dictionary with two entries. The first one, called `predicted_depth`, is a tensor with the values
being the depth expressed in meters for each pixel.
The second one, `depth`, is a PIL image that visualizes the depth estimation result.


key: _node_content
string_value:
{"id_": "58fcf0d4-8ec3-463a-872f-a69cc1713c35", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1adf76d2-39b8-4c8c-8de3-84d00e5bb189", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "44b1eabc695f079866d1f46e61cd6aea2a966b045812ccdec9c9b361190b36af", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4169dcdc-04f0-4578-970c-713a72eb2d90", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "b73189b358261d46b70209cc028fbc6cf01314a3db602144f4dbf65e990e6555", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82d7b026-7751-49da-9409-b63f8520bcb2", "node_type": "1", "metadata": {}, "hash": "60b0ddf4ee66370f637617ab88d293ceaf59c5681e880f8810b1cf490e13453b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1699, "end_char_idx": 4875, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

Check the [Depth Estimation](https://huggingface.co/tasks/depth-estimation) task page to view all compatible architectures and checkpoints.

</Tip>

Before we begin, we need to install the latest version of Transformers:

```bash
pip install -q -U transformers
```

## Depth estimation pipeline

The simplest way to try out inference with a model supporting depth estimation is to use the corresponding [`pipeline`].
Instantiate a pipeline from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads):

```py
>>> from transformers import pipeline
>>> import torch

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> checkpoint = "depth-anything/Depth-Anything-V2-base-hf"
>>> pipe = pipeline("depth-estimation", model=checkpoint, device=device)
```

Next, choose an image to analyze:

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Photo of a bee"/>
</div>

Pass the image to the pipeline.

```py
>>> predictions = pipe(image)
```

The pipeline returns a dictionary with two entries. The first one, called `predicted_depth`, is a tensor with the values
being the depth expressed in meters for each pixel.
The second one, `depth`, is a PIL image that visualizes the depth estimation result.

Let's take a look at the visualized result:

```py
>>> predictions["depth"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/depth-visualization.png" alt="Depth estimation visualization"/>
</div>

## Depth estimation inference by hand

Now that you've seen how to use the depth estimation pipeline, let's see how we can replicate the same result by hand.

Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation

>>> checkpoint = "Intel/zoedepth-nyu-kitti"

>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
>>> model = AutoModelForDepthEstimation.from_pretrained(checkpoint).to(device)
```

Prepare the image input for the model using the `image_processor` that will take care of the necessary image transformations
such as resizing and normalization:

```py
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values.to(device)
```

Pass the prepared inputs through the model:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(pixel_values)
```

Let's post-process and visualize the results. 

We need to pad and then resize the outputs so that predicted depth map has the same dimension as the original image. After resizing we will remove the padded regions from the depth.


key: _node_content
string_value:
{"id_": "82d7b026-7751-49da-9409-b63f8520bcb2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1adf76d2-39b8-4c8c-8de3-84d00e5bb189", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "44b1eabc695f079866d1f46e61cd6aea2a966b045812ccdec9c9b361190b36af", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58fcf0d4-8ec3-463a-872f-a69cc1713c35", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "2faf93440c6c365e155cc19b3b0234ad261651b062a08ef2e4cf3a3b7db3fd43", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "85227f88-b271-43cd-aa97-db8463ce1961", "node_type": "1", "metadata": {}, "hash": "6611cb9536c1c338f1a29f73e6cb075f2cb8c09d8101cad937a6385494e23de0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3081, "end_char_idx": 6301, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The first one, called `predicted_depth`, is a tensor with the values
being the depth expressed in meters for each pixel.
The second one, `depth`, is a PIL image that visualizes the depth estimation result.

Let's take a look at the visualized result:

```py
>>> predictions["depth"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/depth-visualization.png" alt="Depth estimation visualization"/>
</div>

## Depth estimation inference by hand

Now that you've seen how to use the depth estimation pipeline, let's see how we can replicate the same result by hand.

Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation

>>> checkpoint = "Intel/zoedepth-nyu-kitti"

>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
>>> model = AutoModelForDepthEstimation.from_pretrained(checkpoint).to(device)
```

Prepare the image input for the model using the `image_processor` that will take care of the necessary image transformations
such as resizing and normalization:

```py
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values.to(device)
```

Pass the prepared inputs through the model:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(pixel_values)
```

Let's post-process and visualize the results. 

We need to pad and then resize the outputs so that predicted depth map has the same dimension as the original image. After resizing we will remove the padded regions from the depth. 

```py
>>> import numpy as np
>>> import torch.nn.functional as F

>>> predicted_depth = outputs.predicted_depth.unsqueeze(dim=1)
>>> height, width = pixel_values.shape[2:]

>>> height_padding_factor = width_padding_factor = 3
>>> pad_h = int(np.sqrt(height/2) * height_padding_factor)
>>> pad_w = int(np.sqrt(width/2) * width_padding_factor)

>>> if predicted_depth.shape[-2:] != pixel_values.shape[-2:]:
>>>    predicted_depth = F.interpolate(predicted_depth, size= (height, width), mode='bicubic', align_corners=False)

>>> if pad_h > 0:
     predicted_depth = predicted_depth[:, :, pad_h:-pad_h,:]
>>> if pad_w > 0:
     predicted_depth = predicted_depth[:, :, :, pad_w:-pad_w]
```

We can now visualize the results (the function below is taken from the [GaussianObject](https://github.com/GaussianObject/GaussianObject/blob/ad6629efadb57902d5f8bc0fa562258029a4bdf1/pred_monodepth.py#L11) framework).

```py
import matplotlib

def colorize(value, vmin=None, vmax=None, cmap='gray_r', invalid_val=-99, invalid_mask=None, background_color=(128, 128, 128, 255), gamma_corrected=False, value_transform=None):
    """Converts a depth map to a color image.

    Args:
        value (torch.Tensor, numpy.ndarry): Input depth map. Shape: (H, W) or (1, H, W) or (1, 1, H, W). All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used.


key: _node_content
string_value:
{"id_": "85227f88-b271-43cd-aa97-db8463ce1961", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1adf76d2-39b8-4c8c-8de3-84d00e5bb189", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "44b1eabc695f079866d1f46e61cd6aea2a966b045812ccdec9c9b361190b36af", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82d7b026-7751-49da-9409-b63f8520bcb2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "22d64c64686a42fd54424782c6418e114973d4555b10a06954bb7833d3b25e93", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa0ca5f4-18ed-4461-ad4a-c2c4ece20fb5", "node_type": "1", "metadata": {}, "hash": "20999b3db30623ebf30659893e85941c9a515b278ada537140a5baf9b5d985e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4811, "end_char_idx": 7739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
After resizing we will remove the padded regions from the depth. 

```py
>>> import numpy as np
>>> import torch.nn.functional as F

>>> predicted_depth = outputs.predicted_depth.unsqueeze(dim=1)
>>> height, width = pixel_values.shape[2:]

>>> height_padding_factor = width_padding_factor = 3
>>> pad_h = int(np.sqrt(height/2) * height_padding_factor)
>>> pad_w = int(np.sqrt(width/2) * width_padding_factor)

>>> if predicted_depth.shape[-2:] != pixel_values.shape[-2:]:
>>>    predicted_depth = F.interpolate(predicted_depth, size= (height, width), mode='bicubic', align_corners=False)

>>> if pad_h > 0:
     predicted_depth = predicted_depth[:, :, pad_h:-pad_h,:]
>>> if pad_w > 0:
     predicted_depth = predicted_depth[:, :, :, pad_w:-pad_w]
```

We can now visualize the results (the function below is taken from the [GaussianObject](https://github.com/GaussianObject/GaussianObject/blob/ad6629efadb57902d5f8bc0fa562258029a4bdf1/pred_monodepth.py#L11) framework).

```py
import matplotlib

def colorize(value, vmin=None, vmax=None, cmap='gray_r', invalid_val=-99, invalid_mask=None, background_color=(128, 128, 128, 255), gamma_corrected=False, value_transform=None):
    """Converts a depth map to a color image.

    Args:
        value (torch.Tensor, numpy.ndarry): Input depth map. Shape: (H, W) or (1, H, W) or (1, 1, H, W). All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used. Defaults to None.
        vmax (float, optional):  vmax-valued entries are mapped to end color of cmap. If None, value.max() is used. Defaults to None.
        cmap (str, optional): matplotlib colormap to use. Defaults to 'magma_r'.
        invalid_val (int, optional): Specifies value of invalid pixels that should be colored as 'background_color'. Defaults to -99.
        invalid_mask (numpy.ndarray, optional): Boolean mask for invalid regions. Defaults to None.
        background_color (tuple[int], optional): 4-tuple RGB color to give to invalid pixels. Defaults to (128, 128, 128, 255).
        gamma_corrected (bool, optional): Apply gamma correction to colored image. Defaults to False.
        value_transform (Callable, optional): Apply transform function to valid pixels before coloring. Defaults to None.

    Returns:
        numpy.ndarray, dtype - uint8: Colored depth map. Shape: (H, W, 4)
    """
    if isinstance(value, torch.Tensor):
        value = value.detach().cpu().numpy()

    value = value.squeeze()
    if invalid_mask is None:
        invalid_mask = value == invalid_val
    mask = np.logical_not(invalid_mask)

    # normalize
    vmin = np.percentile(value[mask],2) if vmin is None else vmin
    vmax = np.percentile(value[mask],85) if vmax is None else vmax
    if vmin != vmax:
        value = (value - vmin) / (vmax - vmin)  # vmin..vmax
    else:
        # Avoid 0-division
        value = value * 0.


key: _node_content
string_value:
{"id_": "fa0ca5f4-18ed-4461-ad4a-c2c4ece20fb5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1adf76d2-39b8-4c8c-8de3-84d00e5bb189", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "44b1eabc695f079866d1f46e61cd6aea2a966b045812ccdec9c9b361190b36af", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "85227f88-b271-43cd-aa97-db8463ce1961", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "transformers"}, "hash": "7ee612ed24a323070aaa9b5fdf88f29dbeb593e4d7b02899cb7b4d961441fbb7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6148, "end_char_idx": 8680, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used. Defaults to None.
        vmax (float, optional):  vmax-valued entries are mapped to end color of cmap. If None, value.max() is used. Defaults to None.
        cmap (str, optional): matplotlib colormap to use. Defaults to 'magma_r'.
        invalid_val (int, optional): Specifies value of invalid pixels that should be colored as 'background_color'. Defaults to -99.
        invalid_mask (numpy.ndarray, optional): Boolean mask for invalid regions. Defaults to None.
        background_color (tuple[int], optional): 4-tuple RGB color to give to invalid pixels. Defaults to (128, 128, 128, 255).
        gamma_corrected (bool, optional): Apply gamma correction to colored image. Defaults to False.
        value_transform (Callable, optional): Apply transform function to valid pixels before coloring. Defaults to None.

    Returns:
        numpy.ndarray, dtype - uint8: Colored depth map. Shape: (H, W, 4)
    """
    if isinstance(value, torch.Tensor):
        value = value.detach().cpu().numpy()

    value = value.squeeze()
    if invalid_mask is None:
        invalid_mask = value == invalid_val
    mask = np.logical_not(invalid_mask)

    # normalize
    vmin = np.percentile(value[mask],2) if vmin is None else vmin
    vmax = np.percentile(value[mask],85) if vmax is None else vmax
    if vmin != vmax:
        value = (value - vmin) / (vmax - vmin)  # vmin..vmax
    else:
        # Avoid 0-division
        value = value * 0.

    # squeeze last dim if it exists
    # grey out the invalid values

    value[invalid_mask] = np.nan
    cmapper = matplotlib.colormaps.get_cmap(cmap)
    if value_transform:
        value = value_transform(value)
        # value = value / value.max()
    value = cmapper(value, bytes=True)  # (nxmx4)

    # img = value[:, :, :]
    img = value[...]
    img[invalid_mask] = background_color

    #     return img.transpose((2, 0, 1))
    if gamma_corrected:
        # gamma correction
        img = img / 255
        img = np.power(img, 2.2)
        img = img * 255
        img = img.astype(np.uint8)
    return img

>>> result = colorize(predicted_depth.cpu().squeeze().numpy())
>>> Image.fromarray(result)
```



<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/depth-visualization-zoe.png" alt="Depth estimation visualization"/>
</div>


key: _node_content
string_value:
{"id_": "00011be6-9fe1-4d41-974b-d075c8b1a02b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "76febda2-91cf-45d1-b055-39508cf53561", "node_type": "1", "metadata": {}, "hash": "ec76c30556c9af596d7e0c3e842f8117db2d30443eac67efe33c949ddbab7bcd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2858, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Multiple choice

[[open-in-colab]]

A multiple choice task is similar to question answering, except several candidate answers are provided along with a context and the model is trained to select the correct answer.

This guide will show you how to:

1. Finetune [BERT](https://huggingface.co/google-bert/bert-base-uncased) on the `regular` configuration of the [SWAG](https://huggingface.co/datasets/swag) dataset to select the best answer given multiple options and some context.
2. Use your finetuned model for inference.

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load SWAG dataset

Start by loading the `regular` configuration of the SWAG dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> swag = load_dataset("swag", "regular")
```

Then take a look at an example:

```py
>>> swag["train"][0]
{'ending0': 'passes by walking down the street playing their instruments.',
 'ending1': 'has heard approaching them.',
 'ending2': "arrives and they're outside dancing and asleep.",
 'ending3': 'turns the lead singer watches the performance.',
 'fold-ind': '3416',
 'gold-source': 'gold',
 'label': 0,
 'sent1': 'Members of the procession walk down the street holding small horn brass instruments.',
 'sent2': 'A drum line',
 'startphrase': 'Members of the procession walk down the street holding small horn brass instruments. A drum line',
 'video-id': 'anetv_jkn6uvmqwh4'}
```

While it looks like there are a lot of fields here, it is actually pretty straightforward:

- `sent1` and `sent2`: these fields show how a sentence starts, and if you put the two together, you get the `startphrase` field.
- `ending`: suggests a possible ending for how a sentence can end, but only one of them is correct.
- `label`: identifies the correct sentence ending.

## Preprocess

The next step is to load a BERT tokenizer to process the sentence starts and the four possible endings:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
```

The preprocessing function you want to create needs to:

1. Make four copies of the `sent1` field and combine each of them with `sent2` to recreate how a sentence starts.
2. Combine `sent2` with each of the four possible sentence endings.
3. Flatten these two lists so you can tokenize them, and then unflatten them afterward so each example has a corresponding `input_ids`, `attention_mask`, and `labels` field.

```py
>>> ending_names = ["ending0", "ending1", "ending2", "ending3"]


key: _node_content
string_value:
{"id_": "76febda2-91cf-45d1-b055-39508cf53561", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00011be6-9fe1-4d41-974b-d075c8b1a02b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "608961c7eed4d3d5cf0066c2b86117738271075f0ef2a3d5aa5d01d5ca632000", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29910927-07d3-45cc-aca1-b12f3a55a314", "node_type": "1", "metadata": {}, "hash": "a11acb0ccf274d2237be12b7aa9bf4519a33b6b4e53e4ee8993507253230d449", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2861, "end_char_idx": 5965, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def preprocess_function(examples):
...     first_sentences = [[context] * 4 for context in examples["sent1"]]
...     question_headers = examples["sent2"]
...     second_sentences = [
...         [f"{header} {examples[end][i]}" for end in ending_names] for i, header in enumerate(question_headers)
...     ]

...     first_sentences = sum(first_sentences, [])
...     second_sentences = sum(second_sentences, [])

...     tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True)
...     return {k: [v[i : i + 4] for i in range(0, len(v), 4)] for k, v in tokenized_examples.items()}
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
tokenized_swag = swag.map(preprocess_function, batched=True)
```

🤗 Transformers doesn't have a data collator for multiple choice, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

`DataCollatorForMultipleChoice` flattens all the model inputs, applies padding, and then unflattens the results:

<frameworkcontent>
<pt>
```py
>>> from dataclasses import dataclass
>>> from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
>>> from typing import Optional, Union
>>> import torch


>>> @dataclass
... class DataCollatorForMultipleChoice:
...     """
...     Data collator that will dynamically pad the inputs for multiple choice received.
...     """

...     tokenizer: PreTrainedTokenizerBase
...     padding: Union[bool, str, PaddingStrategy] = True
...     max_length: Optional[int] = None
...     pad_to_multiple_of: Optional[int] = None

...     def __call__(self, features):
...         label_name = "label" if "label" in features[0].keys() else "labels"
...         labels = [feature.pop(label_name) for feature in features]
...         batch_size = len(features)
...         num_choices = len(features[0]["input_ids"])
...         flattened_features = [
...             [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features
...         ]
...         flattened_features = sum(flattened_features, [])

...         batch = self.tokenizer.pad(
...             flattened_features,
...             padding=self.padding,
...             max_length=self.max_length,
...             pad_to_multiple_of=self.pad_to_multiple_of,
...             return_tensors="pt",
...         )

...         batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}
...         batch["labels"] = torch.tensor(labels, dtype=torch.int64)
...         return batch
```
</pt>
<tf>
```py
>>> from dataclasses import dataclass
>>> from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
>>> from typing import Optional, Union
>>> import tensorflow as tf


key: _node_content
string_value:
{"id_": "29910927-07d3-45cc-aca1-b12f3a55a314", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "76febda2-91cf-45d1-b055-39508cf53561", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "bf25ec0f767ec49f26d7f01e2b21536db447fc9b016296c3299ea4e389c7bc0d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d0961b44-9bf5-49fe-ac2d-d9b11d3c1b3d", "node_type": "1", "metadata": {}, "hash": "e9d978765075db1b22f1d74c74b718ef61d94a6a8bc634d06b440c420a259fdf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4429, "end_char_idx": 8024, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> @dataclass
... class DataCollatorForMultipleChoice:
...     """
...     Data collator that will dynamically pad the inputs for multiple choice received.
...     """

...     tokenizer: PreTrainedTokenizerBase
...     padding: Union[bool, str, PaddingStrategy] = True
...     max_length: Optional[int] = None
...     pad_to_multiple_of: Optional[int] = None

...     def __call__(self, features):
...         label_name = "label" if "label" in features[0].keys() else "labels"
...         labels = [feature.pop(label_name) for feature in features]
...         batch_size = len(features)
...         num_choices = len(features[0]["input_ids"])
...         flattened_features = [
...             [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features
...         ]
...         flattened_features = sum(flattened_features, [])

...         batch = self.tokenizer.pad(
...             flattened_features,
...             padding=self.padding,
...             max_length=self.max_length,
...             pad_to_multiple_of=self.pad_to_multiple_of,
...             return_tensors="pt",
...         )

...         batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}
...         batch["labels"] = torch.tensor(labels, dtype=torch.int64)
...         return batch
```
</pt>
<tf>
```py
>>> from dataclasses import dataclass
>>> from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
>>> from typing import Optional, Union
>>> import tensorflow as tf


>>> @dataclass
... class DataCollatorForMultipleChoice:
...     """
...     Data collator that will dynamically pad the inputs for multiple choice received.
...     """

...     tokenizer: PreTrainedTokenizerBase
...     padding: Union[bool, str, PaddingStrategy] = True
...     max_length: Optional[int] = None
...     pad_to_multiple_of: Optional[int] = None

...     def __call__(self, features):
...         label_name = "label" if "label" in features[0].keys() else "labels"
...         labels = [feature.pop(label_name) for feature in features]
...         batch_size = len(features)
...         num_choices = len(features[0]["input_ids"])
...         flattened_features = [
...             [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features
...         ]
...         flattened_features = sum(flattened_features, [])

...         batch = self.tokenizer.pad(
...             flattened_features,
...             padding=self.padding,
...             max_length=self.max_length,
...             pad_to_multiple_of=self.pad_to_multiple_of,
...             return_tensors="tf",
...         )

...         batch = {k: tf.reshape(v, (batch_size, num_choices, -1)) for k, v in batch.items()}
...         batch["labels"] = tf.convert_to_tensor(labels, dtype=tf.int64)
...         return batch
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "d0961b44-9bf5-49fe-ac2d-d9b11d3c1b3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29910927-07d3-45cc-aca1-b12f3a55a314", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "4216bf7a99464d29834fd7d80b6d86f828b5dc8b23cb2cb7d67529de9d1efdf2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f1e4001-5631-4697-9c36-0f88db4e94e3", "node_type": "1", "metadata": {}, "hash": "66596a8766692767c9b6d27923e0c7ded80324e4ccd5e9b64dfd8b0773a749e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8027, "end_char_idx": 10511, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load BERT with [`AutoModelForMultipleChoice`]:

```py
>>> from transformers import AutoModelForMultipleChoice, TrainingArguments, Trainer

>>> model = AutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-uncased")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_swag_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_swag["train"],
...     eval_dataset=tokenized_swag["validation"],
...     tokenizer=tokenizer,
...     data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "5f1e4001-5631-4697-9c36-0f88db4e94e3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d0961b44-9bf5-49fe-ac2d-d9b11d3c1b3d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "15fbfe3819d5ef6d3b4bc5e1ca13cdc3b56c80dbdc140310b5b3aaddc0eb0924", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac27eac9-0cfc-40e8-80dd-5c7ebe35d926", "node_type": "1", "metadata": {}, "hash": "236a52b799fa6cd29198d206d0ef024845498ba2a8f422b42b1eb05ba9c54b04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8931, "end_char_idx": 12049, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_swag_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_swag["train"],
...     eval_dataset=tokenized_swag["validation"],
...     tokenizer=tokenizer,
...     data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 2
>>> total_train_steps = (len(tokenized_swag["train"]) // batch_size) * num_train_epochs
>>> optimizer, schedule = create_optimizer(init_lr=5e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load BERT with [`TFAutoModelForMultipleChoice`]:

```py
>>> from transformers import TFAutoModelForMultipleChoice

>>> model = TFAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> data_collator = DataCollatorForMultipleChoice(tokenizer=tokenizer)
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_swag["train"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_swag["validation"],
...     shuffle=False,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> model.compile(optimizer=optimizer)  # No loss argument!


key: _node_content
string_value:
{"id_": "ac27eac9-0cfc-40e8-80dd-5c7ebe35d926", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f1e4001-5631-4697-9c36-0f88db4e94e3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "5d15353e830553f06a19d30ab3c8d490ce12a46e75e704ac54fac0d70ed0b220", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4886b3d0-29a6-4e2d-a9e0-f3a617fd3a0d", "node_type": "1", "metadata": {}, "hash": "124d71ddebd8ec2fb2cb468b4020e45de97ffb9f4d0e01e488b0ea2775438c48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10513, "end_char_idx": 13401, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 2
>>> total_train_steps = (len(tokenized_swag["train"]) // batch_size) * num_train_epochs
>>> optimizer, schedule = create_optimizer(init_lr=5e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load BERT with [`TFAutoModelForMultipleChoice`]:

```py
>>> from transformers import TFAutoModelForMultipleChoice

>>> model = TFAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> data_collator = DataCollatorForMultipleChoice(tokenizer=tokenizer)
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_swag["train"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_swag["validation"],
...     shuffle=False,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=2, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "4886b3d0-29a6-4e2d-a9e0-f3a617fd3a0d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08d762b4-3adb-4976-9101-4dffbbbe601c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "017880e162f4b5b17fe17d1e4e107fee7f1f2a4bb77958970475d05c2faf69c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac27eac9-0cfc-40e8-80dd-5c7ebe35d926", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "transformers"}, "hash": "4f7cb78e44f3e160d2d1959cd7e6b7399cc2da18d4c83d5c714a8c9daace8a8c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11852, "end_char_idx": 15851, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=2, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>


<Tip>

For a more in-depth example of how to finetune a model for multiple choice, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text and two candidate answers:

```py
>>> prompt = "France has a bread law, Le Décret Pain, with strict rules on what is allowed in a traditional baguette."
>>> candidate1 = "The law does not apply to croissants and brioche."
>>> candidate2 = "The law applies to baguettes."
```

<frameworkcontent>
<pt>
Tokenize each prompt and candidate answer pair and return PyTorch tensors. You should also create some `labels`:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_swag_model")
>>> inputs = tokenizer([[prompt, candidate1], [prompt, candidate2]], return_tensors="pt", padding=True)
>>> labels = torch.tensor(0).unsqueeze(0)
```

Pass your inputs and labels to the model and return the `logits`:

```py
>>> from transformers import AutoModelForMultipleChoice

>>> model = AutoModelForMultipleChoice.from_pretrained("my_awesome_swag_model")
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in inputs.items()}, labels=labels)
>>> logits = outputs.logits
```

Get the class with the highest probability:

```py
>>> predicted_class = logits.argmax().item()
>>> predicted_class
'0'
```
</pt>
<tf>
Tokenize each prompt and candidate answer pair and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_swag_model")
>>> inputs = tokenizer([[prompt, candidate1], [prompt, candidate2]], return_tensors="tf", padding=True)
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForMultipleChoice

>>> model = TFAutoModelForMultipleChoice.from_pretrained("my_awesome_swag_model")
>>> inputs = {k: tf.expand_dims(v, 0) for k, v in inputs.items()}
>>> outputs = model(inputs)
>>> logits = outputs.logits
```

Get the class with the highest probability:

```py
>>> predicted_class = int(tf.math.argmax(logits, axis=-1)[0])
>>> predicted_class
'0'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "6d8decf9-34b1-4d00-88ad-7c2317023217", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a0dff318-8d41-4545-9a93-1843e0a6306c", "node_type": "1", "metadata": {}, "hash": "e04c506eb29a179de6d43eac4643a755c26e7f0cc743519bb09c2845305b7e39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question answering

[[open-in-colab]]

<Youtube id="ajPx5LwJD-I"/>

Question answering tasks return an answer given a question. If you've ever asked a virtual assistant like Alexa, Siri or Google what the weather is, then you've used a question answering model before. There are two common types of question answering tasks:

- Extractive: extract the answer from the given context.
- Abstractive: generate an answer from the context that correctly answers the question.

This guide will show you how to:

1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [SQuAD](https://huggingface.co/datasets/squad) dataset for extractive question answering.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/question-answering)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load SQuAD dataset

Start by loading a smaller subset of the SQuAD dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> squad = load_dataset("squad", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> squad = squad.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> squad["train"][0]
{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},
 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
```

There are several important fields here:

- `answers`: the starting location of the answer token and the answer text.
- `context`: background information from which the model needs to extract the answer.
- `question`: the question a model should answer.


key: _node_content
string_value:
{"id_": "a0dff318-8d41-4545-9a93-1843e0a6306c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d8decf9-34b1-4d00-88ad-7c2317023217", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d2f83049344b93da410f966930130c2e19572c8309ef8523d01cb2c686d47b71", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f917a26e-b4a5-4d5f-a448-647b0e76ac28", "node_type": "1", "metadata": {}, "hash": "3d5c739a1ce353b979ff2d97ff8deb958eb9426bc5c6320b404f8d709ae0369c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1509, "end_char_idx": 4025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from datasets import load_dataset

>>> squad = load_dataset("squad", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> squad = squad.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> squad["train"][0]
{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},
 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
```

There are several important fields here:

- `answers`: the starting location of the answer token and the answer text.
- `context`: background information from which the model needs to extract the answer.
- `question`: the question a model should answer.

## Preprocess

<Youtube id="qgaM0weJHpA"/>

The next step is to load a DistilBERT tokenizer to process the `question` and `context` fields:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

There are a few preprocessing steps particular to question answering tasks you should be aware of:

1. Some examples in a dataset may have a very long `context` that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the `context` by setting `truncation="only_second"`.
2. Next, map the start and end positions of the answer to the original `context` by setting
   `return_offset_mapping=True`.
3. With the mapping in hand, now you can find the start and end tokens of the answer. Use the [`~tokenizers.Encoding.sequence_ids`] method to
   find which part of the offset corresponds to the `question` and which corresponds to the `context`.


key: _node_content
string_value:
{"id_": "f917a26e-b4a5-4d5f-a448-647b0e76ac28", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a0dff318-8d41-4545-9a93-1843e0a6306c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "617dbeb042f879f87b56b73453b035c043d7a5ffacbfc85dc360b5fc582c63c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4d74967-5474-4687-91c1-94c9d455c1fd", "node_type": "1", "metadata": {}, "hash": "2f3b76818a8c4b90d8ef8b0e99271d348b4623d0126a1366d5c7318719e5f997", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2367, "end_char_idx": 6117, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
```

There are several important fields here:

- `answers`: the starting location of the answer token and the answer text.
- `context`: background information from which the model needs to extract the answer.
- `question`: the question a model should answer.

## Preprocess

<Youtube id="qgaM0weJHpA"/>

The next step is to load a DistilBERT tokenizer to process the `question` and `context` fields:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

There are a few preprocessing steps particular to question answering tasks you should be aware of:

1. Some examples in a dataset may have a very long `context` that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the `context` by setting `truncation="only_second"`.
2. Next, map the start and end positions of the answer to the original `context` by setting
   `return_offset_mapping=True`.
3. With the mapping in hand, now you can find the start and end tokens of the answer. Use the [`~tokenizers.Encoding.sequence_ids`] method to
   find which part of the offset corresponds to the `question` and which corresponds to the `context`.

Here is how you can create a function to truncate and map the start and end tokens of the `answer` to the `context`:

```py
>>> def preprocess_function(examples):
...     questions = [q.strip() for q in examples["question"]]
...     inputs = tokenizer(
...         questions,
...         examples["context"],
...         max_length=384,
...         truncation="only_second",
...         return_offsets_mapping=True,
...         padding="max_length",
...     )

...     offset_mapping = inputs.pop("offset_mapping")
...     answers = examples["answers"]
...     start_positions = []
...     end_positions = []

...     for i, offset in enumerate(offset_mapping):
...         answer = answers[i]
...         start_char = answer["answer_start"][0]
...         end_char = answer["answer_start"][0] + len(answer["text"][0])
...         sequence_ids = inputs.sequence_ids(i)

...         # Find the start and end of the context
...         idx = 0
...         while sequence_ids[idx] != 1:
...             idx += 1
...         context_start = idx
...         while sequence_ids[idx] == 1:
...             idx += 1
...         context_end = idx - 1

...         # If the answer is not fully inside the context, label it (0, 0)
...         if offset[context_start][0] > end_char or offset[context_end][1] < start_char:
...             start_positions.append(0)
...             end_positions.append(0)
...         else:
...             # Otherwise it's the start and end token positions
...             idx = context_start
...             while idx <= context_end and offset[idx][0] <= start_char:
...                 idx += 1
...             start_positions.append(idx - 1)

...             idx = context_end
...             while idx >= context_start and offset[idx][1] >= end_char:
...                 idx -= 1
...             end_positions.append(idx + 1)

...     inputs["start_positions"] = start_positions
...     inputs["end_positions"] = end_positions
...     return inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function.


key: _node_content
string_value:
{"id_": "c4d74967-5474-4687-91c1-94c9d455c1fd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f917a26e-b4a5-4d5f-a448-647b0e76ac28", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "1fca7f106b0140b26877eb60556b7466e3b53355742e8cb8393dc4841e1497f6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d01c23e-5f3d-435d-a8a5-ad24de457f22", "node_type": "1", "metadata": {}, "hash": "f2632235133c1214f724dcec6853072d824dcae3146ced97f687cdc77f0981e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6118, "end_char_idx": 8779, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once. Remove any columns you don't need:

```py
>>> tokenized_squad = squad.map(preprocess_function, batched=True, remove_columns=squad["train"].column_names)
```

Now create a batch of examples using [`DefaultDataCollator`]. Unlike other data collators in 🤗 Transformers, the [`DefaultDataCollator`] does not apply any additional preprocessing such as padding.

<frameworkcontent>
<pt>
```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```
</pt>
<tf>
```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator(return_tensors="tf")
```
</tf>
</frameworkcontent>

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForQuestionAnswering`]:

```py
>>> from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer

>>> model = AutoModelForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_qa_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_squad["train"],
...     eval_dataset=tokenized_squad["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "9d01c23e-5f3d-435d-a8a5-ad24de457f22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4d74967-5474-4687-91c1-94c9d455c1fd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "8ccdba0217de475f4b01d9a75e1e429a77ce71871088112fd42e654483736a30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "85f38fe4-bfca-46b9-b31d-90b41973eebc", "node_type": "1", "metadata": {}, "hash": "a03f9849ba499f8975613becf42236d30b92d2585f0381d7cc7662419648d4ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7419, "end_char_idx": 10249, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_qa_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_squad["train"],
...     eval_dataset=tokenized_squad["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_epochs = 2
>>> total_train_steps = (len(tokenized_squad["train"]) // batch_size) * num_epochs
>>> optimizer, schedule = create_optimizer(
...     init_lr=2e-5,
...     num_warmup_steps=0,
...     num_train_steps=total_train_steps,
... )
```

Then you can load DistilBERT with [`TFAutoModelForQuestionAnswering`]:

```py
>>> from transformers import TFAutoModelForQuestionAnswering

>>> model = TFAutoModelForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_squad["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_squad["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method):

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)
```

The last thing to setup before you start training is to provide a way to push your model to the Hub.


key: _node_content
string_value:
{"id_": "85f38fe4-bfca-46b9-b31d-90b41973eebc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d01c23e-5f3d-435d-a8a5-ad24de457f22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "1fc4068755746ca17b32691c421cbeab8e89e7e5e5bd51a61b9d6077a1aab74e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "36e4976a-28e4-4014-9e34-046e8b66217d", "node_type": "1", "metadata": {}, "hash": "5633c5115d0f00fe317080a6833cd9457e3c13cb02e8e80f4f639cb8966e7f99", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8781, "end_char_idx": 12050, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_epochs = 2
>>> total_train_steps = (len(tokenized_squad["train"]) // batch_size) * num_epochs
>>> optimizer, schedule = create_optimizer(
...     init_lr=2e-5,
...     num_warmup_steps=0,
...     num_train_steps=total_train_steps,
... )
```

Then you can load DistilBERT with [`TFAutoModelForQuestionAnswering`]:

```py
>>> from transformers import TFAutoModelForQuestionAnswering

>>> model = TFAutoModelForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_squad["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_squad["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method):

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)
```

The last thing to setup before you start training is to provide a way to push your model to the Hub. This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_qa_model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback])
```
Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).

</Tip>

## Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.

If have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the 🤗 Hugging Face Course!

## Inference

Great, now that you've finetuned a model, you can use it for inference!


key: _node_content
string_value:
{"id_": "36e4976a-28e4-4014-9e34-046e8b66217d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "85f38fe4-bfca-46b9-b31d-90b41973eebc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "11abd61fdc01ff27847243c95279f32c9fbc31357cf52d47a2f343543d6a3de2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f124629b-f5ea-4dcb-8e7e-ccc98c625370", "node_type": "1", "metadata": {}, "hash": "e746484cd609606399329f9f411df3abdbc236165789816fc3041c0b37e58bfe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10603, "end_char_idx": 12435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback])
```
Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).

</Tip>

## Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.

If have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the 🤗 Hugging Face Course!

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a question and some context you'd like the model to predict:

```py
>>> question = "How many programming languages does BLOOM support?"
>>> context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`].


key: _node_content
string_value:
{"id_": "f124629b-f5ea-4dcb-8e7e-ccc98c625370", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0644af4b-e7db-4d46-a9bc-a8d41dc816fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "d40ef902670412e44c6eab15348ba95f1cc6da7be2d725be4f054d04caa135cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "36e4976a-28e4-4014-9e34-046e8b66217d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "transformers"}, "hash": "74132e8ba1519a1033c900f4a81915caeca6f5f82e4287f0fff9116c7b84ec6b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10992, "end_char_idx": 15021, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).

</Tip>

## Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.

If have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the 🤗 Hugging Face Course!

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a question and some context you'd like the model to predict:

```py
>>> question = "How many programming languages does BLOOM support?"
>>> context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for question answering with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> question_answerer = pipeline("question-answering", model="my_awesome_qa_model")
>>> question_answerer(question=question, context=context)
{'score': 0.2058267742395401,
 'start': 10,
 'end': 95,
 'answer': '176 billion parameters and can generate text in 46 languages natural languages and 13'}
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_qa_model")
>>> inputs = tokenizer(question, context, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> import torch
>>> from transformers import AutoModelForQuestionAnswering

>>> model = AutoModelForQuestionAnswering.from_pretrained("my_awesome_qa_model")
>>> with torch.no_grad():
...     outputs = model(**inputs)
```

Get the highest probability from the model output for the start and end positions:

```py
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
```

Decode the predicted tokens to get the answer:

```py
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'176 billion parameters and can generate text in 46 languages natural languages and 13'
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_qa_model")
>>> inputs = tokenizer(question, text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForQuestionAnswering

>>> model = TFAutoModelForQuestionAnswering.from_pretrained("my_awesome_qa_model")
>>> outputs = model(**inputs)
```

Get the highest probability from the model output for the start and end positions:

```py
>>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
>>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])
```

Decode the predicted tokens to get the answer:

```py
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'176 billion parameters and can generate text in 46 languages natural languages and 13'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8910169d-f5d6-46fd-9564-3c17dcdb6468", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82d9ee8d-754f-43cf-9d73-73d647a29d85", "node_type": "1", "metadata": {}, "hash": "786ad593ccb8b4a3c4f82237545ba3d694d5f0622a9cadbe988c5ea9379a0904", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3077, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Masked language modeling

[[open-in-colab]]

<Youtube id="mqElG5QJWUg"/>

Masked language modeling predicts a masked token in a sequence, and the model can attend to tokens bidirectionally. This
means the model has full access to the tokens on the left and right. Masked language modeling is great for tasks that
require a good contextual understanding of an entire sequence. BERT is an example of a masked language model.

This guide will show you how to:

1. Finetune [DistilRoBERTa](https://huggingface.co/distilbert/distilroberta-base) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/fill-mask)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load ELI5 dataset

Start by loading the first 5000 examples from the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset with the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?


key: _node_content
string_value:
{"id_": "82d9ee8d-754f-43cf-9d73-73d647a29d85", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8910169d-f5d6-46fd-9564-3c17dcdb6468", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "e1c45bb09fe4bbd82106fcf3f7f521d7146e05b09afb7d89b271ed15ee3f470d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "338043a9-6d87-4820-9e54-8e57088d6de9", "node_type": "1", "metadata": {}, "hash": "fca688bf07822e2d57f1ac5e1f49a2ba723bb41da1cfb7d25f8a8a9b210a2db6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1978, "end_char_idx": 4884, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="8PmhEIXhBvI"/>

For masked language modeling, the next step is to load a DistilRoBERTa tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilroberta-base")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy.


key: _node_content
string_value:
{"id_": "338043a9-6d87-4820-9e54-8e57088d6de9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82d9ee8d-754f-43cf-9d73-73d647a29d85", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9e93fc2b2bcdd25a760aa43c35151ad75215336f3d5ba4a40aa6d070cf76a691", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2040222c-6019-4906-a900-09b18bff1a30", "node_type": "1", "metadata": {}, "hash": "89cb920f142f182cdbb73afb89187d9623f0dcaf139c3ecf7ea62308aad3444f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3508, "end_char_idx": 6534, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
What's cool about language modeling tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="8PmhEIXhBvI"/>

For masked language modeling, the next step is to load a DistilRoBERTa tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilroberta-base")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`.


key: _node_content
string_value:
{"id_": "2040222c-6019-4906-a900-09b18bff1a30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "338043a9-6d87-4820-9e54-8e57088d6de9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "520ef1063244a362f6e7ee0b8732aa6d4834e17a4239847a9e81e96404999c6b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7fe04d43-840d-4bb6-a2fe-9271da77a828", "node_type": "1", "metadata": {}, "hash": "ad41d22d1e30f42d94be711d57f6ce5cf1e72f8e7c9d2f9bc2e73c94392769c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4885, "end_char_idx": 8091, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to
- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. 

```py
>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     return result
```

Apply the `group_texts` function over the entire dataset:

```py
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)
```

Now create a batch of examples using [`DataCollatorForLanguageModeling`].


key: _node_content
string_value:
{"id_": "7fe04d43-840d-4bb6-a2fe-9271da77a828", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2040222c-6019-4906-a900-09b18bff1a30", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "06ea08715d3c26618a18632a6ee32cd737b0233fac6912bb71b5cb6a0eb01fc6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49cb58ba-702a-48d1-907a-5b7038b8b25b", "node_type": "1", "metadata": {}, "hash": "58f107a3b1885214cff3a2ab1f050d40d6500978da4f6cb166f8a6596466ec5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6535, "end_char_idx": 9774, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to
- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. 

```py
>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     return result
```

Apply the `group_texts` function over the entire dataset:

```py
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)
```

Now create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>

Use the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)
```
</pt>
<tf>

Use the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilRoBERTa with [`AutoModelForMaskedLM`]:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).


key: _node_content
string_value:
{"id_": "49cb58ba-702a-48d1-907a-5b7038b8b25b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7fe04d43-840d-4bb6-a2fe-9271da77a828", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "11f4d344258e22141fa192861d40d2a933e22915521886bc9dce98ff12bacd30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c36fb013-51eb-4827-b65c-3c4571171dc3", "node_type": "1", "metadata": {}, "hash": "e141bd8886cdbc40dfed409f0728ad146184a514e3a86ac4a89df08cc96f6911", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8259, "end_char_idx": 10989, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>

Use the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)
```
</pt>
<tf>

Use the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilRoBERTa with [`AutoModelForMaskedLM`]:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_mlm_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:

```py
>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 8.76
```

Then share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "c36fb013-51eb-4827-b65c-3c4571171dc3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49cb58ba-702a-48d1-907a-5b7038b8b25b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "4eb27ff2203f1946019b00a4d0fe987df7f164df6e4149002d14dd54feccc385", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d1e09e1-dc06-4927-ba2e-835c53593956", "node_type": "1", "metadata": {}, "hash": "d9ad6c526f3cdbb490f3b38e09507f8418d81ae632ff0e2baf0144a461ca17c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9497, "end_char_idx": 12288, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_mlm_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:

```py
>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 8.76
```

Then share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilRoBERTa with [`TFAutoModelForMaskedLM`]:

```py
>>> from transformers import TFAutoModelForMaskedLM

>>> model = TFAutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!


key: _node_content
string_value:
{"id_": "2d1e09e1-dc06-4927-ba2e-835c53593956", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c36fb013-51eb-4827-b65c-3c4571171dc3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "39ecce2cf6263810c28d222487d2df58024ccd1fd87aac3e6e3af5283258e53b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be75df2c-ee7f-40cb-aa3a-0b3173a87033", "node_type": "1", "metadata": {}, "hash": "b693c2d971025a507c713a496013b00da3022f285966af65e4247c161de42532", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10991, "end_char_idx": 13911, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilRoBERTa with [`TFAutoModelForMaskedLM`]:

```py
>>> from transformers import TFAutoModelForMaskedLM

>>> model = TFAutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_mlm_model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for masked language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:

```py
>>> text = "The Milky Way is a <mask> galaxy."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it.


key: _node_content
string_value:
{"id_": "be75df2c-ee7f-40cb-aa3a-0b3173a87033", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d1e09e1-dc06-4927-ba2e-835c53593956", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "0a34f9edb7eee16ee69a5ec1a0892d3d3a70de22ddc2dde1f343fa9af2eda5e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "84408b48-cb58-41ad-9410-d9788ed7e543", "node_type": "1", "metadata": {}, "hash": "5b3617d5ea882657a699cfcbadd744546e96406d02d90e3b5947fb7a308b50c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12289, "end_char_idx": 14630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_mlm_model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for masked language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:

```py
>>> text = "The Milky Way is a <mask> galaxy."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it. If you like, you can use the `top_k` parameter to specify how many predictions to return:

```py
>>> from transformers import pipeline

>>> mask_filler = pipeline("fill-mask", "username/my_awesome_eli5_mlm_model")
>>> mask_filler(text, top_k=3)
[{'score': 0.5150994658470154,
  'token': 21300,
  'token_str': ' spiral',
  'sequence': 'The Milky Way is a spiral galaxy.'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors.


key: _node_content
string_value:
{"id_": "84408b48-cb58-41ad-9410-d9788ed7e543", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be75df2c-ee7f-40cb-aa3a-0b3173a87033", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "56dad74a8ba1b3ae1e4e515a550a50675e3174097a1fea659dfc85e8d671c0f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "99c361fe-9524-4f73-86d8-7c42c8f17283", "node_type": "1", "metadata": {}, "hash": "829a88a74a4b39ff287a2f9e7093138bfaf6a7ff78359e5bf2132e5653f0584d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13452, "end_char_idx": 15751, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:

```py
>>> text = "The Milky Way is a <mask> galaxy."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it. If you like, you can use the `top_k` parameter to specify how many predictions to return:

```py
>>> from transformers import pipeline

>>> mask_filler = pipeline("fill-mask", "username/my_awesome_eli5_mlm_model")
>>> mask_filler(text, top_k=3)
[{'score': 0.5150994658470154,
  'token': 21300,
  'token_str': ' spiral',
  'sequence': 'The Milky Way is a spiral galaxy.'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors. You'll also need to specify the position of the `<mask>` token:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="pt")
>>> mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
```

Pass your inputs to the model and return the `logits` of the masked token:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]
```

Then return the three masked tokens with the highest probability and print them out:

```py
>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors.


key: _node_content
string_value:
{"id_": "99c361fe-9524-4f73-86d8-7c42c8f17283", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "442ccb6b-a9d8-40fd-8399-d315c49ca8c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "9b3e8180ef07283ca8562e6514bd7d0c93a63972b09cdd91afad2c57ce3af303", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "84408b48-cb58-41ad-9410-d9788ed7e543", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "transformers"}, "hash": "a25674e94f355fe8a34891f6ced1ef90472df4741160a83d9b0e5c8358257243", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14280, "end_char_idx": 16815, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors. You'll also need to specify the position of the `<mask>` token:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="pt")
>>> mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
```

Pass your inputs to the model and return the `logits` of the masked token:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]
```

Then return the three masked tokens with the highest probability and print them out:

```py
>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors. You'll also need to specify the position of the `<mask>` token:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="tf")
>>> mask_token_index = tf.where(inputs["input_ids"] == tokenizer.mask_token_id)[0, 1]
```

Pass your inputs to the model and return the `logits` of the masked token:

```py
>>> from transformers import TFAutoModelForMaskedLM

>>> model = TFAutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]
```

Then return the three masked tokens with the highest probability and print them out:

```py
>>> top_3_tokens = tf.math.top_k(mask_token_logits, 3).indices.numpy()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "904ebbbe-9880-4148-97eb-ddd3aacfbbc8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "hash": "c18d02b9c0db9f863d39da18348e899e9067197b11d79e1493fcf4c633a0694d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eeebfafe-72c8-4121-9e06-69d22a776728", "node_type": "1", "metadata": {}, "hash": "1ff1cef968511d48382d6cbe7ee3c4b063282962845720e40a5569c0e026c6ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image-to-Image Task Guide

[[open-in-colab]]

Image-to-Image task is the task where an application receives an image and outputs another image. This has various subtasks, including image enhancement (super resolution, low light enhancement, deraining and so on), image inpainting, and more. 

This guide will show you how to:
- Use an image-to-image pipeline for super resolution task,
- Run image-to-image models for same task without a pipeline.

Note that as of the time this guide is released, `image-to-image` pipeline only supports super resolution task.

Let's begin by installing the necessary libraries.

```bash
pip install transformers
```

We can now initialize the pipeline with a [Swin2SR model](https://huggingface.co/caidas/swin2SR-lightweight-x2-64). We can then infer with the pipeline by calling it with an image. As of now, only [Swin2SR models](https://huggingface.co/models?sort=trending&search=swin2sr) are supported in this pipeline. 

```python
from transformers import pipeline

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pipe = pipeline(task="image-to-image", model="caidas/swin2SR-lightweight-x2-64", device=device)
```

Now, let's load an image.

```python
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg"
image = Image.open(requests.get(url, stream=True).raw)

print(image.size)
```
```bash
# (532, 432)
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg" alt="Photo of a cat"/>
</div>

We can now do inference with the pipeline. We will get an upscaled version of the cat image. 

```python
upscaled = pipe(image)
print(upscaled.size)
```
```bash
# (1072, 880)
```

If you wish to do inference yourself with no pipeline, you can use the `Swin2SRForImageSuperResolution` and `Swin2SRImageProcessor` classes of transformers. We will use the same model checkpoint for this. Let's initialize the model and the processor.

```python
from transformers import Swin2SRForImageSuperResolution, Swin2SRImageProcessor 

model = Swin2SRForImageSuperResolution.from_pretrained("caidas/swin2SR-lightweight-x2-64").to(device)
processor = Swin2SRImageProcessor("caidas/swin2SR-lightweight-x2-64")
```

`pipeline` abstracts away the preprocessing and postprocessing steps that we have to do ourselves, so let's preprocess the image. We will pass the image to the processor and then move the pixel values to GPU. 

```python
pixel_values = processor(image, return_tensors="pt").pixel_values
print(pixel_values.shape)

pixel_values = pixel_values.to(device)
```

We can now infer the image by passing pixel values to the model.


key: _node_content
string_value:
{"id_": "eeebfafe-72c8-4121-9e06-69d22a776728", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "hash": "c18d02b9c0db9f863d39da18348e899e9067197b11d79e1493fcf4c633a0694d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "904ebbbe-9880-4148-97eb-ddd3aacfbbc8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "hash": "e84eba29c003a8326201ca7e50761ad69640de84033b84f9cebd7b5a060d3dec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b34f1a36-0f13-4c26-9283-7ab7a25c3396", "node_type": "1", "metadata": {}, "hash": "c28a6859c39eb1a4971ec2bd86ffee8bb3aa9bd5d60ec072f553d0346e80214a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1205, "end_char_idx": 3746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg"
image = Image.open(requests.get(url, stream=True).raw)

print(image.size)
```
```bash
# (532, 432)
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg" alt="Photo of a cat"/>
</div>

We can now do inference with the pipeline. We will get an upscaled version of the cat image. 

```python
upscaled = pipe(image)
print(upscaled.size)
```
```bash
# (1072, 880)
```

If you wish to do inference yourself with no pipeline, you can use the `Swin2SRForImageSuperResolution` and `Swin2SRImageProcessor` classes of transformers. We will use the same model checkpoint for this. Let's initialize the model and the processor.

```python
from transformers import Swin2SRForImageSuperResolution, Swin2SRImageProcessor 

model = Swin2SRForImageSuperResolution.from_pretrained("caidas/swin2SR-lightweight-x2-64").to(device)
processor = Swin2SRImageProcessor("caidas/swin2SR-lightweight-x2-64")
```

`pipeline` abstracts away the preprocessing and postprocessing steps that we have to do ourselves, so let's preprocess the image. We will pass the image to the processor and then move the pixel values to GPU. 

```python
pixel_values = processor(image, return_tensors="pt").pixel_values
print(pixel_values.shape)

pixel_values = pixel_values.to(device)
```

We can now infer the image by passing pixel values to the model.

```python
import torch

with torch.no_grad():
  outputs = model(pixel_values)
```
Output is an object of type `ImageSuperResolutionOutput` that looks like below 👇 

```
(loss=None, reconstruction=tensor([[[[0.8270, 0.8269, 0.8275,  ..., 0.7463, 0.7446, 0.7453],
          [0.8287, 0.8278, 0.8283,  ..., 0.7451, 0.7448, 0.7457],
          [0.8280, 0.8273, 0.8269,  ..., 0.7447, 0.7446, 0.7452],
          ...,
          [0.5923, 0.5933, 0.5924,  ..., 0.0697, 0.0695, 0.0706],
          [0.5926, 0.5932, 0.5926,  ..., 0.0673, 0.0687, 0.0705],
          [0.5927, 0.5914, 0.5922,  ..., 0.0664, 0.0694, 0.0718]]]],
       device='cuda:0'), hidden_states=None, attentions=None)
```
We need to get the `reconstruction` and post-process it for visualization. Let's see how it looks like.

```python
outputs.reconstruction.data.shape
# torch.Size([1, 3, 880, 1072])
```

We need to squeeze the output and get rid of axis 0, clip the values, then convert it to be numpy float.


key: _node_content
string_value:
{"id_": "b34f1a36-0f13-4c26-9283-7ab7a25c3396", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "05348b65-cb37-495e-bb92-f3c0159872a1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "hash": "c18d02b9c0db9f863d39da18348e899e9067197b11d79e1493fcf4c633a0694d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eeebfafe-72c8-4121-9e06-69d22a776728", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "transformers"}, "hash": "3391ce1631deab628406cd98a52706fb957902e2f614dbaf06464bb4de071111", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2780, "end_char_idx": 4405, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import torch

with torch.no_grad():
  outputs = model(pixel_values)
```
Output is an object of type `ImageSuperResolutionOutput` that looks like below 👇 

```
(loss=None, reconstruction=tensor([[[[0.8270, 0.8269, 0.8275,  ..., 0.7463, 0.7446, 0.7453],
          [0.8287, 0.8278, 0.8283,  ..., 0.7451, 0.7448, 0.7457],
          [0.8280, 0.8273, 0.8269,  ..., 0.7447, 0.7446, 0.7452],
          ...,
          [0.5923, 0.5933, 0.5924,  ..., 0.0697, 0.0695, 0.0706],
          [0.5926, 0.5932, 0.5926,  ..., 0.0673, 0.0687, 0.0705],
          [0.5927, 0.5914, 0.5922,  ..., 0.0664, 0.0694, 0.0718]]]],
       device='cuda:0'), hidden_states=None, attentions=None)
```
We need to get the `reconstruction` and post-process it for visualization. Let's see how it looks like.

```python
outputs.reconstruction.data.shape
# torch.Size([1, 3, 880, 1072])
```

We need to squeeze the output and get rid of axis 0, clip the values, then convert it to be numpy float. Then we will arrange axes to have the shape [1072, 880], and finally, bring the output back to range [0, 255].

```python
import numpy as np

# squeeze, take to CPU and clip the values
output = outputs.reconstruction.data.squeeze().cpu().clamp_(0, 1).numpy()
# rearrange the axes
output = np.moveaxis(output, source=0, destination=-1)
# bring values back to pixel values range
output = (output * 255.0).round().astype(np.uint8)
Image.fromarray(output)
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat_upscaled.png" alt="Upscaled photo of a cat"/>
</div>


key: _node_content
string_value:
{"id_": "40465b30-8822-4844-b953-69da0acd1464", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a22188ed-63d2-4995-88a9-b712e114fba8", "node_type": "1", "metadata": {}, "hash": "243206efbd7b313c37e7fbee6b3f6512683901eef66261c089b61a728730135d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2872, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Video classification

[[open-in-colab]]

Video classification is the task of assigning a label or class to an entire video. Videos are expected to have only one class for each video. Video classification models take a video as input and return a prediction about which class the video belongs to. These models can be used to categorize what a video is all about. A real-world application of video classification is action / activity recognition, which is useful for fitness applications. It is also helpful for vision-impaired individuals, especially when they are commuting.

This guide will show you how to:

1. Fine-tune [VideoMAE](https://huggingface.co/docs/transformers/main/en/model_doc/videomae) on a subset of the [UCF101](https://www.crcv.ucf.edu/data/UCF101.php) dataset.
2. Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/video-classification).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q pytorchvideo transformers evaluate
```

You will use [PyTorchVideo](https://pytorchvideo.org/) (dubbed `pytorchvideo`) to process and prepare the videos.

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load UCF101 dataset

Start by loading a subset of the [UCF-101 dataset](https://www.crcv.ucf.edu/data/UCF101.php). This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from huggingface_hub import hf_hub_download

>>> hf_dataset_identifier = "sayakpaul/ucf101-subset"
>>> filename = "UCF101_subset.tar.gz"
>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type="dataset")
```

After the subset has been downloaded, you need to extract the compressed archive:

```py 
>>> import tarfile

>>> with tarfile.open(file_path) as t:
...      t.extractall(".")
```

At a high level, the dataset is organized like so:

```bash
UCF101_subset/
    train/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    val/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    test/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
```

You can then count the number of total videos.


key: _node_content
string_value:
{"id_": "a22188ed-63d2-4995-88a9-b712e114fba8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "40465b30-8822-4844-b953-69da0acd1464", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "2b2f1221b39fa7f1a43310d7c4e67edf7d1c39dbedc5b18702b1e759a8514115", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e28a0280-33f2-4b85-af5b-a6798054b85a", "node_type": "1", "metadata": {}, "hash": "966439955c865a87bec8253a1b82354754a78595997bbe9126fd7f9877ef072a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1259, "end_char_idx": 4199, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load UCF101 dataset

Start by loading a subset of the [UCF-101 dataset](https://www.crcv.ucf.edu/data/UCF101.php). This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from huggingface_hub import hf_hub_download

>>> hf_dataset_identifier = "sayakpaul/ucf101-subset"
>>> filename = "UCF101_subset.tar.gz"
>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type="dataset")
```

After the subset has been downloaded, you need to extract the compressed archive:

```py 
>>> import tarfile

>>> with tarfile.open(file_path) as t:
...      t.extractall(".")
```

At a high level, the dataset is organized like so:

```bash
UCF101_subset/
    train/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    val/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    test/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
```

You can then count the number of total videos.

```py 
>>> import pathlib
>>> dataset_root_path = "UCF101_subset"
>>> dataset_root_path = pathlib.Path(dataset_root_path)
```

```py 
>>> video_count_train = len(list(dataset_root_path.glob("train/*/*.avi")))
>>> video_count_val = len(list(dataset_root_path.glob("val/*/*.avi")))
>>> video_count_test = len(list(dataset_root_path.glob("test/*/*.avi")))
>>> video_total = video_count_train + video_count_val + video_count_test
>>> print(f"Total videos: {video_total}")
```

```py 
>>> all_video_file_paths = (
...     list(dataset_root_path.glob("train/*/*.avi"))
...     + list(dataset_root_path.glob("val/*/*.avi"))
...     + list(dataset_root_path.glob("test/*/*.avi"))
...  )
>>> all_video_file_paths[:5]
```

The (`sorted`) video paths appear like so:

```bash
...
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'
...
```

You will notice that there are video clips belonging to the same group / scene where group is denoted by `g` in the video file paths. `v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.


key: _node_content
string_value:
{"id_": "e28a0280-33f2-4b85-af5b-a6798054b85a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a22188ed-63d2-4995-88a9-b712e114fba8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "604f440e9b9b86622a88fc08bc7abdb2279679621b1b3245e7dbe054d843489f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "184f590e-009e-4cdc-9a0c-d5c3bdfa4270", "node_type": "1", "metadata": {}, "hash": "78251cfd5b51be30a3e650acfea274c93ff6b9873ed11e6aebdadac6a9126842", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2874, "end_char_idx": 5601, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py 
>>> import pathlib
>>> dataset_root_path = "UCF101_subset"
>>> dataset_root_path = pathlib.Path(dataset_root_path)
```

```py 
>>> video_count_train = len(list(dataset_root_path.glob("train/*/*.avi")))
>>> video_count_val = len(list(dataset_root_path.glob("val/*/*.avi")))
>>> video_count_test = len(list(dataset_root_path.glob("test/*/*.avi")))
>>> video_total = video_count_train + video_count_val + video_count_test
>>> print(f"Total videos: {video_total}")
```

```py 
>>> all_video_file_paths = (
...     list(dataset_root_path.glob("train/*/*.avi"))
...     + list(dataset_root_path.glob("val/*/*.avi"))
...     + list(dataset_root_path.glob("test/*/*.avi"))
...  )
>>> all_video_file_paths[:5]
```

The (`sorted`) video paths appear like so:

```bash
...
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'
...
```

You will notice that there are video clips belonging to the same group / scene where group is denoted by `g` in the video file paths. `v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.

For the validation and evaluation splits, you wouldn't want to have video clips from the same group / scene to prevent [data leakage](https://www.kaggle.com/code/alexisbcook/data-leakage). The subset that you are using in this tutorial takes this information into account.

Next up, you will derive the set of labels present in the dataset. Also, create two dictionaries that'll be helpful when initializing the model:

* `label2id`: maps the class names to integers.
* `id2label`: maps the integers to class names. 

```py 
>>> class_labels = sorted({str(path).split("/")[2] for path in all_video_file_paths})
>>> label2id = {label: i for i, label in enumerate(class_labels)}
>>> id2label = {i: label for label, i in label2id.items()}

>>> print(f"Unique classes: {list(label2id.keys())}.")

# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].
```

There are 10 unique classes. For each class, there are 30 videos in the training set.

## Load a model to fine-tune

Instantiate a video classification model from a pretrained checkpoint and its associated image processor. The model's encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset.


key: _node_content
string_value:
{"id_": "184f590e-009e-4cdc-9a0c-d5c3bdfa4270", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e28a0280-33f2-4b85-af5b-a6798054b85a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "a09cf2a703943efa4c473d3d25731756490581b0587cfb0319cb1f39ba5bdf16", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f9592bea-eb1f-4df9-8b05-1a4294b3cbd9", "node_type": "1", "metadata": {}, "hash": "940cdf82eeff2e523ce4798598006d2da23533cc6709c91b06d14dc4e4e8c253", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4120, "end_char_idx": 7403, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.

For the validation and evaluation splits, you wouldn't want to have video clips from the same group / scene to prevent [data leakage](https://www.kaggle.com/code/alexisbcook/data-leakage). The subset that you are using in this tutorial takes this information into account.

Next up, you will derive the set of labels present in the dataset. Also, create two dictionaries that'll be helpful when initializing the model:

* `label2id`: maps the class names to integers.
* `id2label`: maps the integers to class names. 

```py 
>>> class_labels = sorted({str(path).split("/")[2] for path in all_video_file_paths})
>>> label2id = {label: i for i, label in enumerate(class_labels)}
>>> id2label = {i: label for label, i in label2id.items()}

>>> print(f"Unique classes: {list(label2id.keys())}.")

# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].
```

There are 10 unique classes. For each class, there are 30 videos in the training set.

## Load a model to fine-tune

Instantiate a video classification model from a pretrained checkpoint and its associated image processor. The model's encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset.

```py 
>>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification

>>> model_ckpt = "MCG-NJU/videomae-base"
>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
>>> model = VideoMAEForVideoClassification.from_pretrained(
...     model_ckpt,
...     label2id=label2id,
...     id2label=id2label,
...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
... )
```

While the model is loading, you might notice the following warning:

```bash
Some weights of the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']
- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The warning is telling us we are throwing away some weights (e.g. the weights and bias of the `classifier` layer) and randomly initializing some others (the weights and bias of a new `classifier` layer).


key: _node_content
string_value:
{"id_": "f9592bea-eb1f-4df9-8b05-1a4294b3cbd9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "184f590e-009e-4cdc-9a0c-d5c3bdfa4270", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "2b5449a235509cd22fee385fd695487cedc004a606b724f28d123e3255612676", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83458088-32f0-4a91-9093-135015753db2", "node_type": "1", "metadata": {}, "hash": "5344fafcb0b703d191b1fa7053eaaed9e13102a67c39371145655edda82ca724", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5603, "end_char_idx": 9122, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py 
>>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification

>>> model_ckpt = "MCG-NJU/videomae-base"
>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
>>> model = VideoMAEForVideoClassification.from_pretrained(
...     model_ckpt,
...     label2id=label2id,
...     id2label=id2label,
...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
... )
```

While the model is loading, you might notice the following warning:

```bash
Some weights of the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']
- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The warning is telling us we are throwing away some weights (e.g. the weights and bias of the `classifier` layer) and randomly initializing some others (the weights and bias of a new `classifier` layer). This is expected in this case, because we are adding a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.

**Note** that [this checkpoint](https://huggingface.co/MCG-NJU/videomae-base-finetuned-kinetics) leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out [this checkpoint](https://huggingface.co/sayakpaul/videomae-base-finetuned-kinetics-finetuned-ucf101-subset) which was obtained by fine-tuning `MCG-NJU/videomae-base-finetuned-kinetics`.  

## Prepare the datasets for training

For preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. 

```py 
>>> import pytorchvideo.data

>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )

>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
```

For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).


key: _node_content
string_value:
{"id_": "83458088-32f0-4a91-9093-135015753db2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f9592bea-eb1f-4df9-8b05-1a4294b3cbd9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "1ab65c8e0a06b6411fcdbaffd84d464f35ad1a8a94b966ce5370398774ab347a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b900600-5651-4114-82bc-526c2c42e296", "node_type": "1", "metadata": {}, "hash": "3f9ef1f7d715aba10251a8edbfbaa7931a1ef7028a46beeb7b97e7a8c12ae087", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7404, "end_char_idx": 9975, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is expected in this case, because we are adding a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.

**Note** that [this checkpoint](https://huggingface.co/MCG-NJU/videomae-base-finetuned-kinetics) leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out [this checkpoint](https://huggingface.co/sayakpaul/videomae-base-finetuned-kinetics-finetuned-ucf101-subset) which was obtained by fine-tuning `MCG-NJU/videomae-base-finetuned-kinetics`.  

## Prepare the datasets for training

For preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. 

```py 
>>> import pytorchvideo.data

>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )

>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
```

For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).  

Use the `image_processor` associated with the pre-trained model to obtain the following information:

* Image mean and standard deviation with which the video frame pixels will be normalized.
* Spatial resolution to which the video frames will be resized.

Start by defining some constants.

```py
>>> mean = image_processor.image_mean
>>> std = image_processor.image_std
>>> if "shortest_edge" in image_processor.size:
...     height = width = image_processor.size["shortest_edge"]
>>> else:
...     height = image_processor.size["height"]
...     width = image_processor.size["width"]
>>> resize_to = (height, width)

>>> num_frames_to_sample = model.config.num_frames
>>> sample_rate = 4
>>> fps = 30
>>> clip_duration = num_frames_to_sample * sample_rate / fps
```

Now, define the dataset-specific transformations and the datasets respectively.


key: _node_content
string_value:
{"id_": "7b900600-5651-4114-82bc-526c2c42e296", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83458088-32f0-4a91-9093-135015753db2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "d02a707f359c1bc4aed8762981ebd84445ef29eb26e6a3765550541006f12d50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1635f8f4-3fb7-4313-89da-97fd519809ea", "node_type": "1", "metadata": {}, "hash": "1f87221d24048b78cc5c3ea9943f339a06d74fd0edaeb65faff8a10f005e3a9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8095, "end_char_idx": 12062, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Prepare the datasets for training

For preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. 

```py 
>>> import pytorchvideo.data

>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )

>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
```

For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).  

Use the `image_processor` associated with the pre-trained model to obtain the following information:

* Image mean and standard deviation with which the video frame pixels will be normalized.
* Spatial resolution to which the video frames will be resized.

Start by defining some constants.

```py
>>> mean = image_processor.image_mean
>>> std = image_processor.image_std
>>> if "shortest_edge" in image_processor.size:
...     height = width = image_processor.size["shortest_edge"]
>>> else:
...     height = image_processor.size["height"]
...     width = image_processor.size["width"]
>>> resize_to = (height, width)

>>> num_frames_to_sample = model.config.num_frames
>>> sample_rate = 4
>>> fps = 30
>>> clip_duration = num_frames_to_sample * sample_rate / fps
```

Now, define the dataset-specific transformations and the datasets respectively. Starting with the training set: 

```py 
>>> train_transform = Compose(
...     [
...         ApplyTransformToKey(
...             key="video",
...             transform=Compose(
...                 [
...                     UniformTemporalSubsample(num_frames_to_sample),
...                     Lambda(lambda x: x / 255.0),
...                     Normalize(mean, std),
...                     RandomShortSideScale(min_size=256, max_size=320),
...                     RandomCrop(resize_to),
...                     RandomHorizontalFlip(p=0.5),
...                 ]
...             ),
...         ),
...     ]
... )

>>> train_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "train"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("random", clip_duration),
...     decode_audio=False,
...     transform=train_transform,
... )
```

The same sequence of workflow can be applied to the validation and evaluation sets: 

```py 
>>> val_transform = Compose(
...     [
...         ApplyTransformToKey(
...             key="video",
...             transform=Compose(
...                 [
...                     UniformTemporalSubsample(num_frames_to_sample),
...                     Lambda(lambda x: x / 255.0),
...                     Normalize(mean, std),
...                     Resize(resize_to),
...                 ]
...             ),
...         ),
...     ]
... )

>>> val_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "val"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("uniform", clip_duration),
...     decode_audio=False,
...     transform=val_transform,
... )

>>> test_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "test"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("uniform", clip_duration),
...     decode_audio=False,
...     transform=val_transform,
... )
```

**Note**: The above dataset pipelines are taken from the [official PyTorchVideo example](https://pytorchvideo.org/docs/tutorial_classification#dataset).


