# embedding_metadata

key: chroma:document
string_value:
In that case, one should
provide the nodes and corresponding xpaths themselves to the processor, and make sure to set `parse_html` to `False`.

```python
>>> from transformers import MarkupLMProcessor

>>> processor = MarkupLMProcessor.from_pretrained("microsoft/markuplm-base")
>>> processor.parse_html = False

>>> nodes = ["hello", "world", "how", "are"]
>>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span", "html/body", "html/body/div"]
>>> encoding = processor(nodes=nodes, xpaths=xpaths, return_tensors="pt")
>>> print(encoding.keys())
dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])
```

**Use case 3: token classification (training), parse_html=False**

For token classification tasks (such as [SWDE](https://paperswithcode.com/dataset/swde)), one can also provide the
corresponding node labels in order to train a model. The processor will then convert these into token-level `labels`.
By default, it will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the
`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can
initialize the tokenizer with `only_label_first_subword` set to `False`.

```python
>>> from transformers import MarkupLMProcessor

>>> processor = MarkupLMProcessor.from_pretrained("microsoft/markuplm-base")
>>> processor.parse_html = False

>>> nodes = ["hello", "world", "how", "are"]
>>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span", "html/body", "html/body/div"]
>>> node_labels = [1, 2, 2, 1]
>>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt")
>>> print(encoding.keys())
dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq', 'labels'])
```

**Use case 4: web page question answering (inference), parse_html=True**

For question answering tasks on web pages, you can provide a question to the processor. By default, the
processor will use the feature extractor to get all nodes and xpaths, and create [CLS] question tokens [SEP] word tokens [SEP].

```python
>>> from transformers import MarkupLMProcessor

>>> processor = MarkupLMProcessor.from_pretrained("microsoft/markuplm-base")

>>> html_string = """
...  <!DOCTYPE html>
...  <html>
...  <head>
...  <title>Hello world</title>
...  </head>
...  <body>
...  <h1>Welcome</h1>
...  <p>My name is Niels.</p>
...  </body>
...  </html>"""

>>> question = "What's his name?"
>>> encoding = processor(html_string, questions=question, return_tensors="pt")
>>> print(encoding.keys())
dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])
```

**Use case 5: web page question answering (inference), parse_html=False**

For question answering tasks (such as WebSRC), you can provide a question to the processor. If you have extracted
all nodes and xpaths yourself, you can provide them directly to the processor. Make sure to set `parse_html` to `False`.


key: _node_content
string_value:
{"id_": "7a52d7ad-af4f-4570-bdd9-c1743bb33ec0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ba797f60-f9b4-407c-8260-752741094b6c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers"}, "hash": "ec822b4e0adbca794626f831c94dedd7e5c3d4d71ed6c11ae4b90e54e437032d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "968b527e-1cad-427e-b036-7a56303d0486", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/markuplm", "title": "MarkupLM", "tokens": 2673, "retrieve_doc": true, "source": "transformers"}, "hash": "1aaeaed103b7a8bf69e1de3effa3cb4c785e28dad0322427a259417cde01dfdf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7866, "end_char_idx": 10598, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
By default, the
processor will use the feature extractor to get all nodes and xpaths, and create [CLS] question tokens [SEP] word tokens [SEP].

```python
>>> from transformers import MarkupLMProcessor

>>> processor = MarkupLMProcessor.from_pretrained("microsoft/markuplm-base")

>>> html_string = """
...  <!DOCTYPE html>
...  <html>
...  <head>
...  <title>Hello world</title>
...  </head>
...  <body>
...  <h1>Welcome</h1>
...  <p>My name is Niels.</p>
...  </body>
...  </html>"""

>>> question = "What's his name?"
>>> encoding = processor(html_string, questions=question, return_tensors="pt")
>>> print(encoding.keys())
dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])
```

**Use case 5: web page question answering (inference), parse_html=False**

For question answering tasks (such as WebSRC), you can provide a question to the processor. If you have extracted
all nodes and xpaths yourself, you can provide them directly to the processor. Make sure to set `parse_html` to `False`.

```python
>>> from transformers import MarkupLMProcessor

>>> processor = MarkupLMProcessor.from_pretrained("microsoft/markuplm-base")
>>> processor.parse_html = False

>>> nodes = ["hello", "world", "how", "are"]
>>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span", "html/body", "html/body/div"]
>>> question = "What's his name?"
>>> encoding = processor(nodes=nodes, xpaths=xpaths, questions=question, return_tensors="pt")
>>> print(encoding.keys())
dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'xpath_tags_seq', 'xpath_subs_seq'])
```

## Resources

- [Demo notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MarkupLM)
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)

## MarkupLMConfig

[[autodoc]] MarkupLMConfig
    - all

## MarkupLMFeatureExtractor

[[autodoc]] MarkupLMFeatureExtractor
    - __call__

## MarkupLMTokenizer

[[autodoc]] MarkupLMTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## MarkupLMTokenizerFast

[[autodoc]] MarkupLMTokenizerFast
    - all

## MarkupLMProcessor

[[autodoc]] MarkupLMProcessor
    - __call__

## MarkupLMModel

[[autodoc]] MarkupLMModel
    - forward

## MarkupLMForSequenceClassification

[[autodoc]] MarkupLMForSequenceClassification
    - forward

## MarkupLMForTokenClassification

[[autodoc]] MarkupLMForTokenClassification
    - forward

## MarkupLMForQuestionAnswering

[[autodoc]] MarkupLMForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "92db49bf-42a8-43d2-bbe4-1229aa42d01b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9815164-2d69-4796-ab40-04bae4f4e0a0", "node_type": "1", "metadata": {}, "hash": "d3e56931d1a11feb3c87f796a1f24bb508095200f765d983558270273643e5ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3067, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BEiT

## Overview

The BEiT model was proposed in [BEiT: BERT Pre-Training of Image Transformers](https://arxiv.org/abs/2106.08254) by
Hangbo Bao, Li Dong and Furu Wei. Inspired by BERT, BEiT is the first paper that makes self-supervised pre-training of
Vision Transformers (ViTs) outperform supervised pre-training. Rather than pre-training the model to predict the class
of an image (as done in the [original ViT paper](https://arxiv.org/abs/2010.11929)), BEiT models are pre-trained to
predict visual tokens from the codebook of OpenAI's [DALL-E model](https://arxiv.org/abs/2102.12092) given masked
patches.

The abstract from the paper is the following:

*We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation
from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image
modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image
patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into
visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training
objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we
directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder.
Experimental results on image classification and semantic segmentation show that our model achieves competitive results
with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,
significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains
86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was
contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).

## Usage tips

- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They
  outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as
  fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace
  [`ViTFeatureExtractor`] by [`BeitImageProcessor`] and
  [`ViTForImageClassification`] by [`BeitForImageClassification`]).
- There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for
  performing masked image modeling. You can find it [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BEiT).


key: _node_content
string_value:
{"id_": "a9815164-2d69-4796-ab40-04bae4f4e0a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92db49bf-42a8-43d2-bbe4-1229aa42d01b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6257c64012a687ef28534ef3f5d7472681d437bd78e4e011ad0b1b84a67a7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e6ec7322-38b1-47a7-9ea1-0061d70f16eb", "node_type": "1", "metadata": {}, "hash": "3a81ece81201e5b44a5db6112158defb294e7caedc0eab21dfaeebe5df6d13cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1640, "end_char_idx": 4555, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,
significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains
86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The JAX/FLAX version of this model was
contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/beit).

## Usage tips

- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They
  outperform both the [original model (ViT)](vit) as well as [Data-efficient Image Transformers (DeiT)](deit) when fine-tuned on ImageNet-1K and CIFAR-100. You can check out demo notebooks regarding inference as well as
  fine-tuning on custom data [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer) (you can just replace
  [`ViTFeatureExtractor`] by [`BeitImageProcessor`] and
  [`ViTForImageClassification`] by [`BeitForImageClassification`]).
- There's also a demo notebook available which showcases how to combine DALL-E's image tokenizer with BEiT for
  performing masked image modeling. You can find it [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BEiT).
- As the BEiT models expect each image to be of the same size (resolution), one can use
  [`BeitImageProcessor`] to resize (or rescale) and normalize images for the model.
- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of
  each checkpoint. For example, `microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch
  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=microsoft/beit).
- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of
  14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million
  images and 1,000 classes).
- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the
  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position
  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to
  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the
  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add
  position embeddings.


key: _node_content
string_value:
{"id_": "e6ec7322-38b1-47a7-9ea1-0061d70f16eb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9815164-2d69-4796-ab40-04bae4f4e0a0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "c6caa421c927a35787624f2129f6db7b32f04d2e0d8f9b7979b10d119556582f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7074ae17-ae43-4ac0-acfa-894ae2ee9865", "node_type": "1", "metadata": {}, "hash": "383aea2d91e4685f986ab066d0b16b34fe375ff09ef83b09016af3dc98ca770f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3068, "end_char_idx": 5696, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- As the BEiT models expect each image to be of the same size (resolution), one can use
  [`BeitImageProcessor`] to resize (or rescale) and normalize images for the model.
- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of
  each checkpoint. For example, `microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch
  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=microsoft/beit).
- The available checkpoints are either (1) pre-trained on [ImageNet-22k](http://www.image-net.org/) (a collection of
  14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on [ImageNet-1k](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million
  images and 1,000 classes).
- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the
  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position
  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to
  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the
  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add
  position embeddings.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/beit_architecture.jpg"
alt="drawing" width="600"/>

<small> BEiT pre-training. Taken from the <a href="https://arxiv.org/abs/2106.08254">original paper.</a> </small>

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BEiT.

<PipelineTag pipeline="image-classification"/>

- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

**Semantic segmentation**
- [Semantic segmentation task guide](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "7074ae17-ae43-4ac0-acfa-894ae2ee9865", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cd47ccf3-f986-4664-8f78-cb604e270efe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "1329ad921b76a360c9c50f07d4cd99eccf192477fbcb2faf8d3dcdca2c08ddbc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e6ec7322-38b1-47a7-9ea1-0061d70f16eb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/beit", "title": "BEiT", "tokens": 1832, "retrieve_doc": true, "source": "transformers"}, "hash": "a09b2578565fa327e02964582e2ba12d88ce22c94f5c7c22bf5f3268b0de0a47", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4004, "end_char_idx": 6752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the
  relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position
  bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to
  pre-train a model from scratch, one needs to either set the `use_relative_position_bias` or the
  `use_relative_position_bias` attribute of [`BeitConfig`] to `True` in order to add
  position embeddings.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/beit_architecture.jpg"
alt="drawing" width="600"/>

<small> BEiT pre-training. Taken from the <a href="https://arxiv.org/abs/2106.08254">original paper.</a> </small>

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BEiT.

<PipelineTag pipeline="image-classification"/>

- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

**Semantic segmentation**
- [Semantic segmentation task guide](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## BEiT specific outputs

[[autodoc]] models.beit.modeling_beit.BeitModelOutputWithPooling

[[autodoc]] models.beit.modeling_flax_beit.FlaxBeitModelOutputWithPooling

## BeitConfig

[[autodoc]] BeitConfig

## BeitFeatureExtractor

[[autodoc]] BeitFeatureExtractor
    - __call__
    - post_process_semantic_segmentation

## BeitImageProcessor

[[autodoc]] BeitImageProcessor
    - preprocess
    - post_process_semantic_segmentation

<frameworkcontent>
<pt>

## BeitModel

[[autodoc]] BeitModel
    - forward

## BeitForMaskedImageModeling

[[autodoc]] BeitForMaskedImageModeling
    - forward

## BeitForImageClassification

[[autodoc]] BeitForImageClassification
    - forward

## BeitForSemanticSegmentation

[[autodoc]] BeitForSemanticSegmentation
    - forward

</pt>
<jax>

## FlaxBeitModel

[[autodoc]] FlaxBeitModel
    - __call__

## FlaxBeitForMaskedImageModeling

[[autodoc]] FlaxBeitForMaskedImageModeling
    - __call__

## FlaxBeitForImageClassification

[[autodoc]] FlaxBeitForImageClassification
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "9aa5518f-277b-42a3-b258-9715fda3dc66", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roc_bert", "title": "RoCBert", "tokens": 843, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c9a17ac-536f-461d-b031-a3ab37e3f8ce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roc_bert", "title": "RoCBert", "tokens": 843, "retrieve_doc": true, "source": "transformers"}, "hash": "29a2c7b768273c705aa7bbe6a595c6d18c24302b6a97371e897724b0777a6855", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2771, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoCBert

## Overview

The RoCBert model was proposed in [RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining](https://aclanthology.org/2022.acl-long.65.pdf)  by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou.
It's a pretrained Chinese language model that is robust under various forms of adversarial attacks.

The abstract from the paper is the following:

*Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown
vulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose
ROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation,
synonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency
under different synthesized adversarial examples. The model takes as input multimodal information including the
semantic, phonetic and visual features. We show all these features are important to the model robustness since the
attack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT outperforms strong baselines under
three blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best
in the toxic content detection task under human-made attacks.*

This model was contributed by [weiweishi](https://huggingface.co/weiweishi).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## RoCBertConfig

[[autodoc]] RoCBertConfig
    - all

## RoCBertTokenizer

[[autodoc]] RoCBertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## RoCBertModel

[[autodoc]] RoCBertModel
    - forward

## RoCBertForPreTraining

[[autodoc]] RoCBertForPreTraining
    - forward

## RoCBertForCausalLM

[[autodoc]] RoCBertForCausalLM
    - forward

## RoCBertForMaskedLM

[[autodoc]] RoCBertForMaskedLM
    - forward

## RoCBertForSequenceClassification

[[autodoc]] transformers.RoCBertForSequenceClassification
    - forward

## RoCBertForMultipleChoice

[[autodoc]] transformers.RoCBertForMultipleChoice
    - forward

## RoCBertForTokenClassification

[[autodoc]] transformers.RoCBertForTokenClassification
    - forward

## RoCBertForQuestionAnswering

[[autodoc]] RoCBertForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "61e17414-8f2e-44b8-a3b4-2234eb9ecc50", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/swiftformer", "title": "SwiftFormer", "tokens": 774, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e611041-a433-45a5-8b5d-0063384d3300", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/swiftformer", "title": "SwiftFormer", "tokens": 774, "retrieve_doc": true, "source": "transformers"}, "hash": "a5d9bc408c2943dc09dc397aa8c00bd3a5e2a5fb6e387b674d59f1a10a4a0a40", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SwiftFormer

## Overview

The SwiftFormer model was proposed in [SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications](https://arxiv.org/abs/2303.15446) by Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan.

The SwiftFormer paper introduces a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations in the self-attention computation with linear element-wise multiplications. A series of models called 'SwiftFormer' is built based on this, which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Even their small variant achieves 78.5% top-1 ImageNet1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2× faster compared to MobileViT-v2.

The abstract from the paper is the following:

*Self-attention has become a defacto choice for capturing global context in various vision applications. However, its quadratic computational complexity with respect to image resolution limits its use in real-time applications, especially for deployment on resource-constrained mobile devices. Although hybrid approaches have been proposed to combine the advantages of convolutions and self-attention for a better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. In this work, we introduce a novel efficient additive attention mechanism that effectively replaces the quadratic matrix multiplication operations with linear element-wise multiplications. Our design shows that the key-value interaction can be replaced with a linear layer without sacrificing any accuracy. Unlike previous state-of-the-art methods, our efficient formulation of self-attention enables its usage at all stages of the network. Using our proposed efficient additive attention, we build a series of models called "SwiftFormer" which achieves state-of-the-art performance in terms of both accuracy and mobile inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster compared to MobileViT-v2.*

This model was contributed by [shehan97](https://huggingface.co/shehan97). The TensorFlow version was contributed by [joaocmd](https://huggingface.co/joaocmd).
The original code can be found [here](https://github.com/Amshaker/SwiftFormer).

## SwiftFormerConfig

[[autodoc]] SwiftFormerConfig

## SwiftFormerModel

[[autodoc]] SwiftFormerModel
    - forward

## SwiftFormerForImageClassification

[[autodoc]] SwiftFormerForImageClassification
    - forward

## TFSwiftFormerModel

[[autodoc]] TFSwiftFormerModel
    - call

## TFSwiftFormerForImageClassification

[[autodoc]] TFSwiftFormerForImageClassification
    - call


key: _node_content
string_value:
{"id_": "e9197f4c-9eb9-480b-85e1-a908d08756c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b07c43ff-6602-46da-a4a3-ab5c3251164a", "node_type": "1", "metadata": {}, "hash": "d9383a3826a20c50a2c23a9d66c5773e749623bac8849a6c1402d3447bbd68c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3651, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SeamlessM4T-v2

## Overview

The SeamlessM4T-v2 model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.

SeamlessM4T-v2 is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text. It is an improvement on the [previous version](https://huggingface.co/docs/transformers/main/model_doc/seamless_m4t). For more details on the differences between v1 and v2, refer to section [Difference with SeamlessM4T-v1](#difference-with-seamlessm4t-v1).

SeamlessM4T-v2 enables multiple tasks without relying on separate models:

- Speech-to-speech translation (S2ST)
- Speech-to-text translation (S2TT)
- Text-to-speech translation (T2ST)
- Text-to-text translation (T2TT)
- Automatic speech recognition (ASR)

[`SeamlessM4Tv2Model`] can perform all the above tasks, but each task also has its own dedicated sub-model.

The abstract from the paper is the following:

*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model—SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one’s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time.


key: _node_content
string_value:
{"id_": "b07c43ff-6602-46da-a4a3-ab5c3251164a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9197f4c-9eb9-480b-85e1-a908d08756c9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "539d35860868f918b8be0b56a5e51753bea3cff3bac26fb4538bf7585a567744", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a50d8c2a-862f-4a64-b2d0-def847a2ead5", "node_type": "1", "metadata": {}, "hash": "325a8ef556a08a12499522197b9defa003d2d2842b57cbc64fa2af0d72006f30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1669, "end_char_idx": 5084, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model—SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one’s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work—including models, code, and a watermark detector—are publicly released and accessible at the link below.*

## Usage

In the following example, we'll load an Arabic audio sample and an English text sample and convert them into Russian speech and French text.

First, load the processor and a checkpoint of the model:

```python
>>> from transformers import AutoProcessor, SeamlessM4Tv2Model

>>> processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
>>> model = SeamlessM4Tv2Model.from_pretrained("facebook/seamless-m4t-v2-large")
```

You can seamlessly use this model on text or on audio, to generated either translated text or translated audio.

Here is how to use the processor to process text and audio:

```python
>>> # let's load an audio sample from an Arabic speech corpus
>>> from datasets import load_dataset
>>> dataset = load_dataset("arabic_speech_corpus", split="test", streaming=True)
>>> audio_sample = next(iter(dataset))["audio"]

>>> # now, process it
>>> audio_inputs = processor(audios=audio_sample["array"], return_tensors="pt")

>>> # now, process some English text as well
>>> text_inputs = processor(text = "Hello, my dog is cute", src_lang="eng", return_tensors="pt")
```


key: _node_content
string_value:
{"id_": "a50d8c2a-862f-4a64-b2d0-def847a2ead5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b07c43ff-6602-46da-a4a3-ab5c3251164a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "eb8f8e04ca46f92288b9e4bc0570d45b3535967ee4543485022d8b06d1b602b9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ef5161d-1eb1-401f-af04-630eee19a5d0", "node_type": "1", "metadata": {}, "hash": "4fa031276ab73fc812a1556270b865cb0bca0c8aa5377fb2dc0ae98808c78b04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3437, "end_char_idx": 6469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work—including models, code, and a watermark detector—are publicly released and accessible at the link below.*

## Usage

In the following example, we'll load an Arabic audio sample and an English text sample and convert them into Russian speech and French text.

First, load the processor and a checkpoint of the model:

```python
>>> from transformers import AutoProcessor, SeamlessM4Tv2Model

>>> processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
>>> model = SeamlessM4Tv2Model.from_pretrained("facebook/seamless-m4t-v2-large")
```

You can seamlessly use this model on text or on audio, to generated either translated text or translated audio.

Here is how to use the processor to process text and audio:

```python
>>> # let's load an audio sample from an Arabic speech corpus
>>> from datasets import load_dataset
>>> dataset = load_dataset("arabic_speech_corpus", split="test", streaming=True)
>>> audio_sample = next(iter(dataset))["audio"]

>>> # now, process it
>>> audio_inputs = processor(audios=audio_sample["array"], return_tensors="pt")

>>> # now, process some English text as well
>>> text_inputs = processor(text = "Hello, my dog is cute", src_lang="eng", return_tensors="pt")
```


### Speech

[`SeamlessM4Tv2Model`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:

```python
>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
```

With basically the same code, I've translated English text and Arabic speech to Russian speech samples.

### Text

Similarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4Tv2Model.generate`].
This time, let's translate to French.

```python 
>>> # from audio
>>> output_tokens = model.generate(**audio_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)

>>> # from text
>>> output_tokens = model.generate(**text_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)
```

### Tips


#### 1. Use dedicated models

[`SeamlessM4Tv2Model`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.


key: _node_content
string_value:
{"id_": "9ef5161d-1eb1-401f-af04-630eee19a5d0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a50d8c2a-862f-4a64-b2d0-def847a2ead5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "ea4419864f8cf8cd666d8dd7c911f1dff2d7762bd26045c18e49ecebeccc14b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82a16217-0075-44bd-ad27-38316242328e", "node_type": "1", "metadata": {}, "hash": "7a5fa43e44f764b39e5975ea825b2b3123485478864f4081c62a8953a4c2a6cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5087, "end_char_idx": 8385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Speech

[`SeamlessM4Tv2Model`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:

```python
>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
```

With basically the same code, I've translated English text and Arabic speech to Russian speech samples.

### Text

Similarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4Tv2Model.generate`].
This time, let's translate to French.

```python 
>>> # from audio
>>> output_tokens = model.generate(**audio_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)

>>> # from text
>>> output_tokens = model.generate(**text_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)
```

### Tips


#### 1. Use dedicated models

[`SeamlessM4Tv2Model`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.
For example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: 

```python
>>> from transformers import SeamlessM4Tv2ForSpeechToSpeech
>>> model = SeamlessM4Tv2ForSpeechToSpeech.from_pretrained("facebook/seamless-m4t-v2-large")
```

Or you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.

```python
>>> from transformers import SeamlessM4Tv2ForTextToText
>>> model = SeamlessM4Tv2ForTextToText.from_pretrained("facebook/seamless-m4t-v2-large")
```

Feel free to try out [`SeamlessM4Tv2ForSpeechToText`] and [`SeamlessM4Tv2ForTextToSpeech`] as well.

#### 2. Change the speaker identity

You have the possibility to change the speaker used for speech synthesis with the `speaker_id` argument. Some `speaker_id` works better than other for some languages!

#### 3. Change the generation strategy

You can use different [generation strategies](../generation_strategies) for text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, text_do_sample=True)` which will perform multinomial beam-search decoding on the text model. Note that speech generation only supports greedy - by default - or multinomial sampling, which can be used with e.g. `.generate(..., speech_do_sample=True, speech_temperature=0.6)`.

#### 4. Generate speech and text at the same time

Use `return_intermediate_token_ids=True` with [`SeamlessM4Tv2Model`] to return both speech and text !

## Model architecture

SeamlessM4T-v2 features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as "unit tokens," from the translated text.


key: _node_content
string_value:
{"id_": "82a16217-0075-44bd-ad27-38316242328e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ef5161d-1eb1-401f-af04-630eee19a5d0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "b761eb8710fc563499cd5b4d68bf4798f4838e9f6e0a192189cd4814c38e2245", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9509091c-015b-4a43-8144-ef125c9927f0", "node_type": "1", "metadata": {}, "hash": "1930bd56ca3593f025900111e8145bad1e17ab45ace2f0dcf739e1bddd907cab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6933, "end_char_idx": 10436, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> from transformers import SeamlessM4Tv2ForTextToText
>>> model = SeamlessM4Tv2ForTextToText.from_pretrained("facebook/seamless-m4t-v2-large")
```

Feel free to try out [`SeamlessM4Tv2ForSpeechToText`] and [`SeamlessM4Tv2ForTextToSpeech`] as well.

#### 2. Change the speaker identity

You have the possibility to change the speaker used for speech synthesis with the `speaker_id` argument. Some `speaker_id` works better than other for some languages!

#### 3. Change the generation strategy

You can use different [generation strategies](../generation_strategies) for text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, text_do_sample=True)` which will perform multinomial beam-search decoding on the text model. Note that speech generation only supports greedy - by default - or multinomial sampling, which can be used with e.g. `.generate(..., speech_do_sample=True, speech_temperature=0.6)`.

#### 4. Generate speech and text at the same time

Use `return_intermediate_token_ids=True` with [`SeamlessM4Tv2Model`] to return both speech and text !

## Model architecture

SeamlessM4T-v2 features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as "unit tokens," from the translated text.

Each modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.

### Difference with SeamlessM4T-v1

The architecture of this new version differs from the first in a few aspects:

#### Improvements on the second-pass model

The second seq2seq model, named text-to-unit model, is now non-auto regressive, meaning that it computes units in a **single forward pass**. This achievement is made possible by:
- the use of **character-level embeddings**, meaning that each character of the predicted translated text has its own embeddings, which are then used to predict the unit tokens.
- the use of an intermediate duration predictor, that predicts speech duration at the **character-level** on the predicted translated text.
- the use of a new text-to-unit decoder mixing convolutions and self-attention to handle longer context.

#### Difference in the speech encoder

The speech encoder, which is used during the first-pass generation process to predict the translated text, differs mainly from the previous speech encoder through these mechanisms:
- the use of chunked attention mask to prevent attention across chunks, ensuring that each position attends only to positions within its own chunk and a fixed number of previous chunks.
- the use of relative position embeddings which only considers distance between sequence elements rather than absolute positions. Please refer to [Self-Attentionwith Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155) for more details.
- the use of a causal depth-wise convolution instead of a non-causal one.

### Generation process

Here's how the generation process works:

- Input text or speech is processed through its specific encoder.
- A decoder creates text tokens in the desired language.
- If speech generation is required, the second seq2seq model, generates unit tokens in an non auto-regressive way.


key: _node_content
string_value:
{"id_": "9509091c-015b-4a43-8144-ef125c9927f0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0bf3ef-b81a-4a32-8c6f-5ae79741c01d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "37fca48a2508dd873d46a86c895976af883f282a000c596c1db84d196a8ed78e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82a16217-0075-44bd-ad27-38316242328e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2", "title": "SeamlessM4T-v2", "tokens": 2580, "retrieve_doc": true, "source": "transformers"}, "hash": "b0930f04b1126154ea8dd5221b6e36f87fd077fbd5917bdc161359a1c559bb44", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8623, "end_char_idx": 11248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Difference with SeamlessM4T-v1

The architecture of this new version differs from the first in a few aspects:

#### Improvements on the second-pass model

The second seq2seq model, named text-to-unit model, is now non-auto regressive, meaning that it computes units in a **single forward pass**. This achievement is made possible by:
- the use of **character-level embeddings**, meaning that each character of the predicted translated text has its own embeddings, which are then used to predict the unit tokens.
- the use of an intermediate duration predictor, that predicts speech duration at the **character-level** on the predicted translated text.
- the use of a new text-to-unit decoder mixing convolutions and self-attention to handle longer context.

#### Difference in the speech encoder

The speech encoder, which is used during the first-pass generation process to predict the translated text, differs mainly from the previous speech encoder through these mechanisms:
- the use of chunked attention mask to prevent attention across chunks, ensuring that each position attends only to positions within its own chunk and a fixed number of previous chunks.
- the use of relative position embeddings which only considers distance between sequence elements rather than absolute positions. Please refer to [Self-Attentionwith Relative Position Representations (Shaw et al.)](https://arxiv.org/abs/1803.02155) for more details.
- the use of a causal depth-wise convolution instead of a non-causal one.

### Generation process

Here's how the generation process works:

- Input text or speech is processed through its specific encoder.
- A decoder creates text tokens in the desired language.
- If speech generation is required, the second seq2seq model, generates unit tokens in an non auto-regressive way.
- These unit tokens are then passed through the final vocoder to produce the actual speech.


This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## SeamlessM4Tv2Model

[[autodoc]] SeamlessM4Tv2Model
    - generate


## SeamlessM4Tv2ForTextToSpeech

[[autodoc]] SeamlessM4Tv2ForTextToSpeech
    - generate


## SeamlessM4Tv2ForSpeechToSpeech

[[autodoc]] SeamlessM4Tv2ForSpeechToSpeech
    - generate


## SeamlessM4Tv2ForTextToText

[[autodoc]] transformers.SeamlessM4Tv2ForTextToText
    - forward
    - generate

## SeamlessM4Tv2ForSpeechToText

[[autodoc]] transformers.SeamlessM4Tv2ForSpeechToText
    - forward
    - generate

## SeamlessM4Tv2Config

[[autodoc]] SeamlessM4Tv2Config


key: _node_content
string_value:
{"id_": "65623014-e540-43f0-8615-e40468d9ea0c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "hash": "3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c674773-6177-49de-a7e1-c4357e6728ae", "node_type": "1", "metadata": {}, "hash": "aa255320c91e0db463d18ac19d0834b0ae60749832dfa14119603bb8193913da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ViTMSN

## Overview

The ViTMSN model was proposed in [Masked Siamese Networks for Label-Efficient Learning](https://arxiv.org/abs/2204.07141) by Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes,
Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas. The paper presents a joint-embedding architecture to match the prototypes
of masked patches with that of the unmasked patches. With this setup, their method yields excellent performance in the low-shot and extreme low-shot
regimes.

The abstract from the paper is the following:

*We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our
approach matches the representation of an image view containing randomly masked patches to the representation of the original
unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the
unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures,
while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance,
on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy,
and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark.*

<img src="https://i.ibb.co/W6PQMdC/Screenshot-2022-09-13-at-9-08-40-AM.png" alt="drawing" width="600"/> 

<small> MSN architecture. Taken from the <a href="https://arxiv.org/abs/2204.07141">original paper.</a> </small>

This model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/facebookresearch/msn). 

## Usage tips

- MSN (masked siamese networks) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training
objective is to match the prototypes assigned to the unmasked views of the images to that of the masked views of the same images.
- The authors have only released pre-trained weights of the backbone (ImageNet-1k pre-training). So, to use that on your own image classification dataset,
use the [`ViTMSNForImageClassification`] class which is initialized from [`ViTMSNModel`]. Follow
[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.
- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K
labels when fine-tuned.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use.


key: _node_content
string_value:
{"id_": "5c674773-6177-49de-a7e1-c4357e6728ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "hash": "3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65623014-e540-43f0-8615-e40468d9ea0c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "hash": "f5940b0f1fd37c3b961c35d8f0643586bc1087079573dbe966f8c9254e97d34b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e753cd48-3f44-4d64-a22d-c0cc7c3e37e5", "node_type": "1", "metadata": {}, "hash": "f18e90a610b2c2bf2562ac4e6bb1b4e04c4470ecc99d5f7d7d1d92a7709b5604", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1452, "end_char_idx": 3904, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

<img src="https://i.ibb.co/W6PQMdC/Screenshot-2022-09-13-at-9-08-40-AM.png" alt="drawing" width="600"/> 

<small> MSN architecture. Taken from the <a href="https://arxiv.org/abs/2204.07141">original paper.</a> </small>

This model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/facebookresearch/msn). 

## Usage tips

- MSN (masked siamese networks) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training
objective is to match the prototypes assigned to the unmasked views of the images to that of the masked views of the same images.
- The authors have only released pre-trained weights of the backbone (ImageNet-1k pre-training). So, to use that on your own image classification dataset,
use the [`ViTMSNForImageClassification`] class which is initialized from [`ViTMSNModel`]. Follow
[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.
- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K
labels when fine-tuned.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ViTMSNForImageClassification
model = ViTMSNForImageClassification.from_pretrained("facebook/vit-msn-base", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-msn-base` model, we saw the following speedups during inference.


key: _node_content
string_value:
{"id_": "e753cd48-3f44-4d64-a22d-c0cc7c3e37e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "efbf1f99-5bcc-416a-a4a3-8ca60647a679", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "hash": "3190299d5fce31c7d896ea5b4190690ffc9b3c8c3d4d5769212bf67361cd974b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5c674773-6177-49de-a7e1-c4357e6728ae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_msn", "title": "ViTMSN", "tokens": 1465, "retrieve_doc": true, "source": "transformers"}, "hash": "a42f440f1be22aa93224a5a436e55275441ce2799536995499343d1737158d14", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2349, "end_char_idx": 5668, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Follow
[this notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb) for a detailed tutorial on fine-tuning.
- MSN is particularly useful in the low-shot and extreme low-shot regimes. Notably, it achieves 75.7% top-1 accuracy with only 1% of ImageNet-1K
labels when fine-tuned.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ViTMSNForImageClassification
model = ViTMSNForImageClassification.from_pretrained("facebook/vit-msn-base", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/vit-msn-base` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         7 |                                         6 |                      1.17 |
|            2 |                                         8 |                                         6 |                      1.33 |
|            4 |                                         8 |                                         6 |                      1.33 |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT MSN.

<PipelineTag pipeline="image-classification"/>

- [`ViTMSNForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ViTMSNConfig

[[autodoc]] ViTMSNConfig

## ViTMSNModel

[[autodoc]] ViTMSNModel
    - forward

## ViTMSNForImageClassification

[[autodoc]] ViTMSNForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "778078ac-c723-496a-b3ec-b6d7bd950574", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "hash": "e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5729b037-fecb-4f13-9ad7-62a0bfef1023", "node_type": "1", "metadata": {}, "hash": "2f6bedb1c220372b107306358a0681d3b21299a7c7f5a177d8900ea26200c0a6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3482, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OneFormer

## Overview

The OneFormer model was proposed in [OneFormer: One Transformer to Rule Universal Image Segmentation](https://arxiv.org/abs/2211.06220) by Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi. OneFormer is a universal image segmentation framework that can be trained on a single panoptic dataset to perform semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference.

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_teaser.png"/>

The abstract from the paper is the following:

*Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.*

The figure below illustrates the architecture of OneFormer. Taken from the [original paper](https://arxiv.org/abs/2211.06220).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_architecture.png"/>

This model was contributed by [Jitesh Jain](https://huggingface.co/praeclarumjj3). The original code can be found [here](https://github.com/SHI-Labs/OneFormer).

## Usage tips

-  OneFormer requires two inputs during inference: *image* and *task token*. 
- During training, OneFormer only uses panoptic annotations.
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  `get_num_masks` function inside in the `OneFormerLoss` class of `modeling_oneformer.py`. When training on multiple nodes, this should be
  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).


key: _node_content
string_value:
{"id_": "5729b037-fecb-4f13-9ad7-62a0bfef1023", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "hash": "e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "778078ac-c723-496a-b3ec-b6d7bd950574", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "hash": "fb99f3051e6bb3bcf462cdb3bf4b6626efaf567cf080eb68977e4836a4b8d5ed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "75f47238-1434-40f0-a58c-70f6a62fd579", "node_type": "1", "metadata": {}, "hash": "f1b33ed8f1a7972823247100188125b82de29ee031ecf2af4efc41f862f06e5a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1815, "end_char_idx": 4766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.*

The figure below illustrates the architecture of OneFormer. Taken from the [original paper](https://arxiv.org/abs/2211.06220).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/oneformer_architecture.png"/>

This model was contributed by [Jitesh Jain](https://huggingface.co/praeclarumjj3). The original code can be found [here](https://github.com/SHI-Labs/OneFormer).

## Usage tips

-  OneFormer requires two inputs during inference: *image* and *task token*. 
- During training, OneFormer only uses panoptic annotations.
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  `get_num_masks` function inside in the `OneFormerLoss` class of `modeling_oneformer.py`. When training on multiple nodes, this should be
  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).
- One can use [`OneFormerProcessor`] to prepare input images and task inputs for the model and optional targets for the model. [`OneformerProcessor`] wraps [`OneFormerImageProcessor`] and [`CLIPTokenizer`] into a single instance to both prepare the images and encode the task inputs.
- To get the final segmentation, depending on the task, you can call [`~OneFormerProcessor.post_process_semantic_segmentation`] or [`~OneFormerImageProcessor.post_process_instance_segmentation`] or [`~OneFormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`OneFormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with OneFormer.

- Demo notebooks regarding inference + fine-tuning on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OneFormer).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "75f47238-1434-40f0-a58c-70f6a62fd579", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "91271b29-e5e4-4e37-9ae0-3c35641e33a1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "hash": "e25644105ec2643710c168684894bbc3a9e2bf1f03240f4133eb2e748ec8217b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5729b037-fecb-4f13-9ad7-62a0bfef1023", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/oneformer", "title": "OneFormer", "tokens": 1357, "retrieve_doc": true, "source": "transformers"}, "hash": "939a1c7e988168925c49b6d2d1d15560056dc37d89e4db4a9aef4c76cf16a72d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3197, "end_char_idx": 5445, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When training on multiple nodes, this should be
  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/SHI-Labs/OneFormer/blob/33ebb56ed34f970a30ae103e786c0cb64c653d9a/oneformer/modeling/criterion.py#L287).
- One can use [`OneFormerProcessor`] to prepare input images and task inputs for the model and optional targets for the model. [`OneformerProcessor`] wraps [`OneFormerImageProcessor`] and [`CLIPTokenizer`] into a single instance to both prepare the images and encode the task inputs.
- To get the final segmentation, depending on the task, you can call [`~OneFormerProcessor.post_process_semantic_segmentation`] or [`~OneFormerImageProcessor.post_process_instance_segmentation`] or [`~OneFormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`OneFormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with OneFormer.

- Demo notebooks regarding inference + fine-tuning on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OneFormer).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.

## OneFormer specific outputs

[[autodoc]] models.oneformer.modeling_oneformer.OneFormerModelOutput

[[autodoc]] models.oneformer.modeling_oneformer.OneFormerForUniversalSegmentationOutput

## OneFormerConfig

[[autodoc]] OneFormerConfig

## OneFormerImageProcessor

[[autodoc]] OneFormerImageProcessor
    - preprocess
    - encode_inputs
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## OneFormerProcessor

[[autodoc]] OneFormerProcessor

## OneFormerModel

[[autodoc]] OneFormerModel
    - forward

## OneFormerForUniversalSegmentation

[[autodoc]] OneFormerForUniversalSegmentation
    - forward


key: _node_content
string_value:
{"id_": "b87e35e2-5d9d-48de-97ea-9767a64c92a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sew", "title": "SEW", "tokens": 629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a1ec8e58-c5b8-4a82-ba89-c2b775adb4eb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sew", "title": "SEW", "tokens": 629, "retrieve_doc": true, "source": "transformers"}, "hash": "48c93e7aacb4789c8c0d458461eafea0471b5599c273e6d4b3cedc401536e322", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1811, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SEW

## Overview

SEW (Squeezed and Efficient Wav2Vec) was proposed in [Performance-Efficiency Trade-offs in Unsupervised Pre-training
for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q.
Weinberger, Yoav Artzi.

The abstract from the paper is the following:

*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition
(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance
and its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a
pre-trained model architecture with significant improvements along both performance and efficiency dimensions across a
variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x
inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference
time, SEW reduces word error rate by 25-50% across different model sizes.*

This model was contributed by [anton-l](https://huggingface.co/anton-l).

## Usage tips

- SEW is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- SEWForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded using
  [`Wav2Vec2CTCTokenizer`].

## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## SEWConfig

[[autodoc]] SEWConfig

## SEWModel

[[autodoc]] SEWModel
    - forward

## SEWForCTC

[[autodoc]] SEWForCTC
    - forward

## SEWForSequenceClassification

[[autodoc]] SEWForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "99181b5d-8740-4f9a-a448-43e8f47227bf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers"}, "hash": "7f5f33c1273d0ef1283fb7b8cb8974be2aa8c1405422beaed3e59413eb0eafd8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d1cde9b-8436-4bf5-bba0-0be9f862f205", "node_type": "1", "metadata": {}, "hash": "592ccbb7cfd3ec4ff63797fb6a80e4e7dd13e4036ff74e872b8fd85bb8c09661", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2996, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# AltCLIP

## Overview

The AltCLIP model was proposed in [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://arxiv.org/abs/2211.06679v2) by Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, Ledell Wu. AltCLIP
(Altering the Language Encoder in CLIP) is a neural network trained on a variety of image-text and text-text pairs. By switching CLIP's
text encoder with a pretrained multilingual text encoder XLM-R, we could obtain very close performances with CLIP on almost all tasks, and extended original CLIP's capabilities such as multilingual understanding.

The abstract from the paper is the following:

*In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. 
Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained 
multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of 
teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art 
performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with 
CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*

This model was contributed by [jongjyh](https://huggingface.co/jongjyh).

## Usage tips and example

The usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention
and we take the [CLS] token in XLM-R to represent text embedding.

AltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image
classification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text
features. Both the text and visual features are then projected to a latent space with identical dimension. The dot
product between the projected image and text features is then used as a similar score.

To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,
which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors
also add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.
The [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.

The [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokenizer`] into a single instance to both
encode the text and prepare the images. The following example shows how to get the image-text similarity scores using
[`AltCLIPProcessor`] and [`AltCLIPModel`].


key: _node_content
string_value:
{"id_": "9d1cde9b-8436-4bf5-bba0-0be9f862f205", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "69c7558e-5c4c-4a47-bad9-61284865bdaa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers"}, "hash": "7f5f33c1273d0ef1283fb7b8cb8974be2aa8c1405422beaed3e59413eb0eafd8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "99181b5d-8740-4f9a-a448-43e8f47227bf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/altclip", "title": "AltCLIP", "tokens": 1185, "retrieve_doc": true, "source": "transformers"}, "hash": "b1ea86842c272b8e69db36ecbb603c80d80ba84983bf0300d3c4aaa1d0bbce47", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1286, "end_char_idx": 4294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Further, we obtain very close performances with 
CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding.*

This model was contributed by [jongjyh](https://huggingface.co/jongjyh).

## Usage tips and example

The usage of AltCLIP is very similar to the CLIP. the difference between CLIP is the text encoder. Note that we use bidirectional attention instead of casual attention
and we take the [CLS] token in XLM-R to represent text embedding.

AltCLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image
classification. AltCLIP uses a ViT like transformer to get visual features and a bidirectional language model to get the text
features. Both the text and visual features are then projected to a latent space with identical dimension. The dot
product between the projected image and text features is then used as a similar score.

To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,
which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors
also add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.
The [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.

The [`AltCLIPProcessor`] wraps a [`CLIPImageProcessor`] and a [`XLMRobertaTokenizer`] into a single instance to both
encode the text and prepare the images. The following example shows how to get the image-text similarity scores using
[`AltCLIPProcessor`] and [`AltCLIPModel`].

```python
>>> from PIL import Image
>>> import requests

>>> from transformers import AltCLIPModel, AltCLIPProcessor

>>> model = AltCLIPModel.from_pretrained("BAAI/AltCLIP")
>>> processor = AltCLIPProcessor.from_pretrained("BAAI/AltCLIP")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
```

<Tip>

This model is based on `CLIPModel`, use it like you would use the original [CLIP](clip).

</Tip>

## AltCLIPConfig

[[autodoc]] AltCLIPConfig
    - from_text_vision_configs

## AltCLIPTextConfig

[[autodoc]] AltCLIPTextConfig

## AltCLIPVisionConfig

[[autodoc]] AltCLIPVisionConfig

## AltCLIPProcessor

[[autodoc]] AltCLIPProcessor

## AltCLIPModel

[[autodoc]] AltCLIPModel
    - forward
    - get_text_features
    - get_image_features

## AltCLIPTextModel

[[autodoc]] AltCLIPTextModel
    - forward

## AltCLIPVisionModel

[[autodoc]] AltCLIPVisionModel
    - forward


key: _node_content
string_value:
{"id_": "1b1c04d0-7ba3-43b8-b265-f5fc1ace166f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers"}, "hash": "bedc083f2570c825df7e8f8b4f61b04ab2a997667c2ed1cbe90abd7b6c28d376", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88bb438e-d994-4dc4-a234-2171507802b6", "node_type": "1", "metadata": {}, "hash": "1c0a9a223d9d212987dc2f2a45726c2147d1b52cf1f64272755adcf7d43a0848", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3094, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PaliGemma

## Overview

The PaliGemma model was proposed in [PaliGemma – Google's Cutting-Edge Open Vision Language Model](https://huggingface.co/blog/paligemma) by Google. It is a 3B vision-language model composed by a [SigLIP](siglip) vision encoder and a [Gemma](gemma) language decoder linked by a multimodal linear projection. It cuts an image into a fixed number of VIT tokens and prepends it to an optional prompt. One particularity is that the model uses full block attention on all the image tokens plus the input text tokens. It comes in 3 resolutions, 224x224, 448x448 and 896x896 with 3 base models, with 55 fine-tuned versions for different tasks, and 2 mix models.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/paligemma/paligemma_arch.png"
alt="drawing" width="600"/>

<small> PaliGemma architecture. Taken from the <a href="https://huggingface.co/blog/paligemma">blog post.</a> </small>

This model was contributed by [Molbap](https://huggingface.co/Molbap).

## Usage tips

Inference with PaliGemma can be performed as follows:

```python
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration

model_id = "google/paligemma-3b-mix-224"
model = PaliGemmaForConditionalGeneration.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

prompt = "What is on the flower?"
image_file = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=true"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=20)

print(processor.decode(output[0], skip_special_tokens=True)[len(prompt):])
```

- PaliGemma is not meant for conversational use, and it works best when fine-tuning to a specific use case. Some downstream tasks on which PaliGemma can be fine-tuned include image captioning, visual question answering (VQA), object detection, referring expression segmentation and document understanding.
- One can use `PaliGemmaProcessor` to prepare images, text and optional labels for the model. When fine-tuning a PaliGemma model, the `suffix` argument can be passed to the processor which creates the `labels` for the model:

```python
prompt = "What is on the flower?"
answer = "a bee"
inputs = processor(text=prompt, images=raw_image, suffix=answer, return_tensors="pt")
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with PaliGemma. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A blog post introducing all the features of PaliGemma can be found [here](https://huggingface.co/blog/paligemma).
- Demo notebooks on how to fine-tune PaliGemma for VQA with the Trainer API along with inference can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/paligemma).


key: _node_content
string_value:
{"id_": "88bb438e-d994-4dc4-a234-2171507802b6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f780dda-f343-41c4-a076-136ed706b2ad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers"}, "hash": "bedc083f2570c825df7e8f8b4f61b04ab2a997667c2ed1cbe90abd7b6c28d376", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1b1c04d0-7ba3-43b8-b265-f5fc1ace166f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/paligemma", "title": "PaliGemma", "tokens": 1039, "retrieve_doc": true, "source": "transformers"}, "hash": "29b13b939a85a7a340c15623c2a63500b3448408487d054540e59efd59a851fb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1859, "end_char_idx": 3508, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some downstream tasks on which PaliGemma can be fine-tuned include image captioning, visual question answering (VQA), object detection, referring expression segmentation and document understanding.
- One can use `PaliGemmaProcessor` to prepare images, text and optional labels for the model. When fine-tuning a PaliGemma model, the `suffix` argument can be passed to the processor which creates the `labels` for the model:

```python
prompt = "What is on the flower?"
answer = "a bee"
inputs = processor(text=prompt, images=raw_image, suffix=answer, return_tensors="pt")
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with PaliGemma. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A blog post introducing all the features of PaliGemma can be found [here](https://huggingface.co/blog/paligemma).
- Demo notebooks on how to fine-tune PaliGemma for VQA with the Trainer API along with inference can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/paligemma).
- Demo notebooks on how to fine-tune PaliGemma on a custom dataset (receipt image -> JSON) along with inference can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/PaliGemma). 🌎

## PaliGemmaConfig

[[autodoc]] PaliGemmaConfig

## PaliGemmaProcessor

[[autodoc]] PaliGemmaProcessor

## PaliGemmaForConditionalGeneration

[[autodoc]] PaliGemmaForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "0d69273f-bb56-4013-b6df-70cad2a920cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f26ea05d-07b4-4ca8-824c-30806ce02fbe", "node_type": "1", "metadata": {}, "hash": "b95a7e96913cb57cce5d38f405fab9a95088afff62ccc00a0ec909cc4d1f2278", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3392, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Encoder Decoder Models

## Overview

The [`EncoderDecoderModel`] can be used to initialize a sequence-to-sequence model with any
pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder.

The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks
was shown in [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://arxiv.org/abs/1907.12461) by
Sascha Rothe, Shashi Narayan, Aliaksei Severyn.

After such an [`EncoderDecoderModel`] has been trained/fine-tuned, it can be saved/loaded just like
any other models (see the examples for more information).

An application of this architecture could be to leverage two pretrained [`BertModel`] as the encoder
and decoder for a summarization model as was shown in: [Text Summarization with Pretrained Encoders](https://arxiv.org/abs/1908.08345) by Yang Liu and Mirella Lapata.

## Randomly initializing `EncoderDecoderModel` from model configurations.

[`EncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`BertModel`] configuration for the encoder and the default [`BertForCausalLM`] configuration for the decoder.

```python
>>> from transformers import BertConfig, EncoderDecoderConfig, EncoderDecoderModel

>>> config_encoder = BertConfig()
>>> config_decoder = BertConfig()

>>> config = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
>>> model = EncoderDecoderModel(config=config)
```

## Initialising `EncoderDecoderModel` from a pretrained encoder and a pretrained decoder.

[`EncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, *e.g.* BERT, can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing [`EncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).
To do so, the `EncoderDecoderModel` class provides a [`EncoderDecoderModel.from_encoder_decoder_pretrained`] method.

```python
>>> from transformers import EncoderDecoderModel, BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("google-bert/bert-base-uncased", "google-bert/bert-base-uncased")
```

## Loading an existing `EncoderDecoderModel` checkpoint and perform inference.

To load fine-tuned checkpoints of the `EncoderDecoderModel` class, [`EncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.

To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.


key: _node_content
string_value:
{"id_": "f26ea05d-07b4-4ca8-824c-30806ce02fbe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d69273f-bb56-4013-b6df-70cad2a920cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "e88bb2fdb3af0bb37f9240ce62105bb947bfe5a76ba187dedfec987f2496b910", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1e15aa7a-5fdf-4d24-8d77-f27e513e174d", "node_type": "1", "metadata": {}, "hash": "b852675c1986f54dcd15c5d45b997ca5592027b501a1171722a88a4ea9ceabea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1692, "end_char_idx": 4961, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`EncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained auto-encoding model, *e.g.* BERT, can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing [`EncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).
To do so, the `EncoderDecoderModel` class provides a [`EncoderDecoderModel.from_encoder_decoder_pretrained`] method.

```python
>>> from transformers import EncoderDecoderModel, BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("google-bert/bert-base-uncased", "google-bert/bert-base-uncased")
```

## Loading an existing `EncoderDecoderModel` checkpoint and perform inference.

To load fine-tuned checkpoints of the `EncoderDecoderModel` class, [`EncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.

To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.

```python
>>> from transformers import AutoTokenizer, EncoderDecoderModel

>>> # load a fine-tuned seq2seq model and corresponding tokenizer
>>> model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
>>> tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")

>>> # let's perform inference on a long piece of text
>>> ARTICLE_TO_SUMMARIZE = (
...     "PG&E stated it scheduled the blackouts in response to forecasts for high winds "
...     "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were "
...     "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."
... )
>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors="pt").input_ids

>>> # autoregressively generate summary (uses greedy decoding by default)
>>> generated_ids = model.generate(input_ids)
>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
nearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.
```

## Loading a PyTorch checkpoint into `TFEncoderDecoderModel`.

[`TFEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a
pytorch checkpoint. Passing `from_pt=True` to this method will throw an exception.


key: _node_content
string_value:
{"id_": "1e15aa7a-5fdf-4d24-8d77-f27e513e174d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f26ea05d-07b4-4ca8-824c-30806ce02fbe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "6e1fc1bdfb6bf9a1d70d26b1b45639cb40fe00216303fe7ea5b7b3d641afb613", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a92776fb-4829-411e-bc83-aae3eb9c7827", "node_type": "1", "metadata": {}, "hash": "e5c66299667eee86a06fd473c110aeb101ff6e5a7e8525ef41ce2ec6f8d6674d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3187, "end_char_idx": 6570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.

```python
>>> from transformers import AutoTokenizer, EncoderDecoderModel

>>> # load a fine-tuned seq2seq model and corresponding tokenizer
>>> model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
>>> tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")

>>> # let's perform inference on a long piece of text
>>> ARTICLE_TO_SUMMARIZE = (
...     "PG&E stated it scheduled the blackouts in response to forecasts for high winds "
...     "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were "
...     "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."
... )
>>> input_ids = tokenizer(ARTICLE_TO_SUMMARIZE, return_tensors="pt").input_ids

>>> # autoregressively generate summary (uses greedy decoding by default)
>>> generated_ids = model.generate(input_ids)
>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
nearly 800 thousand customers were affected by the shutoffs. the aim is to reduce the risk of wildfires. nearly 800, 000 customers were expected to be affected by high winds amid dry conditions. pg & e said it scheduled the blackouts to last through at least midday tomorrow.
```

## Loading a PyTorch checkpoint into `TFEncoderDecoderModel`.

[`TFEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a
pytorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only pytorch
checkpoints for a particular encoder-decoder model, a workaround is:

```python
>>> # a workaround to load from pytorch checkpoint
>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel

>>> _model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert-cnn_dailymail-fp16")

>>> _model.encoder.save_pretrained("./encoder")
>>> _model.decoder.save_pretrained("./decoder")

>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "./encoder", "./decoder", encoder_from_pt=True, decoder_from_pt=True
... )
>>> # This is only for copying some specific attributes of this particular model.
>>> model.config = _model.config
```

## Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.
As you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the
`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded
target sequence).

```python
>>> from transformers import BertTokenizer, EncoderDecoderModel

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("google-bert/bert-base-uncased", "google-bert/bert-base-uncased")

>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
>>> model.config.pad_token_id = tokenizer.pad_token_id

>>> input_ids = tokenizer(
...     "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris.


key: _node_content
string_value:
{"id_": "a92776fb-4829-411e-bc83-aae3eb9c7827", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1e15aa7a-5fdf-4d24-8d77-f27e513e174d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "2cdaca6044c3864aa4509ca7a978528f02a7dcfebff13f4cabb3a971f28a9f97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6bfadcc0-9529-4e83-805f-9f3b32958858", "node_type": "1", "metadata": {}, "hash": "c0f072ed16a5400a026af7494c78925cda08ed3c4904b3455cfc26033194c93d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4899, "end_char_idx": 8077, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Passing `from_pt=True` to this method will throw an exception. If there are only pytorch
checkpoints for a particular encoder-decoder model, a workaround is:

```python
>>> # a workaround to load from pytorch checkpoint
>>> from transformers import EncoderDecoderModel, TFEncoderDecoderModel

>>> _model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert-cnn_dailymail-fp16")

>>> _model.encoder.save_pretrained("./encoder")
>>> _model.decoder.save_pretrained("./decoder")

>>> model = TFEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "./encoder", "./decoder", encoder_from_pt=True, decoder_from_pt=True
... )
>>> # This is only for copying some specific attributes of this particular model.
>>> model.config = _model.config
```

## Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model.
As you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the
`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded
target sequence).

```python
>>> from transformers import BertTokenizer, EncoderDecoderModel

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained("google-bert/bert-base-uncased", "google-bert/bert-base-uncased")

>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
>>> model.config.pad_token_id = tokenizer.pad_token_id

>>> input_ids = tokenizer(
...     "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.",
...     return_tensors="pt",
... ).input_ids

>>> labels = tokenizer(
...     "the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.",
...     return_tensors="pt",
... ).input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
```

Detailed [colab](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=ZwQIEhKOrJpl) for training.

This model was contributed by [thomwolf](https://github.com/thomwolf). This model's TensorFlow and Flax versions
were contributed by [ydshieh](https://github.com/ydshieh).


key: _node_content
string_value:
{"id_": "6bfadcc0-9529-4e83-805f-9f3b32958858", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bc5ec3f-5258-4ca0-8ceb-305c9fac2ad0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "e2efdaa1bfb1a39545221ba209d082233a8c938ece0da1cbc4b100f3bb8b46f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a92776fb-4829-411e-bc83-aae3eb9c7827", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encoder-decoder", "title": "Encoder Decoder Models", "tokens": 2117, "retrieve_doc": true, "source": "transformers"}, "hash": "b9a801f70aefeed36f908428af1ea393cf905cd2115111554ae05922f37a5509", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6571, "end_char_idx": 8552, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Its base is square, measuring 125 metres (410 ft) on each side.During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was  finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.",
...     return_tensors="pt",
... ).input_ids

>>> labels = tokenizer(
...     "the eiffel tower surpassed the washington monument to become the tallest structure in the world. it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris.",
...     return_tensors="pt",
... ).input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
```

Detailed [colab](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=ZwQIEhKOrJpl) for training.

This model was contributed by [thomwolf](https://github.com/thomwolf). This model's TensorFlow and Flax versions
were contributed by [ydshieh](https://github.com/ydshieh).


## EncoderDecoderConfig

[[autodoc]] EncoderDecoderConfig

<frameworkcontent>
<pt>

## EncoderDecoderModel

[[autodoc]] EncoderDecoderModel
    - forward
    - from_encoder_decoder_pretrained

</pt>
<tf>

## TFEncoderDecoderModel

[[autodoc]] TFEncoderDecoderModel
    - call
    - from_encoder_decoder_pretrained

</tf>
<jax>

## FlaxEncoderDecoderModel

[[autodoc]] FlaxEncoderDecoderModel
    - __call__
    - from_encoder_decoder_pretrained

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "6938536c-6a02-47d0-a228-5a8ec2947b0a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ef21361-6443-413f-a86e-5d9bda19ad5d", "node_type": "1", "metadata": {}, "hash": "25d2a0215c0df4b5d8ed28cb3447787b102f66870704c54698f3aba2359e5b68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2731, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SigLIP

## Overview

The SigLIP model was proposed in [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343) by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. SigLIP proposes to replace the loss function used in [CLIP](clip) by a simple pairwise sigmoid loss. This results in better performance in terms of zero-shot classification accuracy on ImageNet.

The abstract from the paper is the following:

*We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.*

## Usage tips

- Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax.
- Training is supported but does not use `torch.distributed` utilities which may limit the scalability of batch size. However, DDP and FDSP works on single-node multi-gpu setup.
- When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding="max_length"` as that's how the model was trained.
- To get the same results as the pipeline, a prompt template of "This is a photo of {label}." should be used.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg"
alt="drawing" width="600"/>

<small> SigLIP evaluation results compared to CLIP. Taken from the <a href="https://arxiv.org/abs/2303.15343">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/big_vision/tree/main).

## Usage example

There are 2 main ways to use SigLIP: either using the pipeline API, which abstracts away all the complexity for you, or by using the `SiglipModel` class yourself.


key: _node_content
string_value:
{"id_": "1ef21361-6443-413f-a86e-5d9bda19ad5d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6938536c-6a02-47d0-a228-5a8ec2947b0a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "b4ea411b6c4134b543ce30c25ad92ce9c44dc20963413fba7159a3b8c03e5513", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "46c8aec0-1503-4f90-8b7c-f4e993986f16", "node_type": "1", "metadata": {}, "hash": "c1dda6edc01caeedb43185c9921a9034699c21f9fc8a60bdcf49a7dbb5bd1311", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1151, "end_char_idx": 4239, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.*

## Usage tips

- Usage of SigLIP is similar to [CLIP](clip). The main difference is the training loss, which does not require a global view of all the pairwise similarities of images and texts within a batch. One needs to apply the sigmoid activation function to the logits, rather than the softmax.
- Training is supported but does not use `torch.distributed` utilities which may limit the scalability of batch size. However, DDP and FDSP works on single-node multi-gpu setup.
- When using the standalone [`SiglipTokenizer`] or [`SiglipProcessor`], make sure to pass `padding="max_length"` as that's how the model was trained.
- To get the same results as the pipeline, a prompt template of "This is a photo of {label}." should be used.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg"
alt="drawing" width="600"/>

<small> SigLIP evaluation results compared to CLIP. Taken from the <a href="https://arxiv.org/abs/2303.15343">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/big_vision/tree/main).

## Usage example

There are 2 main ways to use SigLIP: either using the pipeline API, which abstracts away all the complexity for you, or by using the `SiglipModel` class yourself.

### Pipeline API

The pipeline allows to use the model in a few lines of code:

```python
>>> from transformers import pipeline
>>> from PIL import Image
>>> import requests

>>> # load pipe
>>> image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-224")

>>> # load image
>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> # inference
>>> candidate_labels = ["2 cats", "a plane", "a remote"]
>>> outputs = image_classifier(image, candidate_labels=candidate_labels)
>>> outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
>>> print(outputs)
[{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}]
```

### Using the model yourself

If you want to do the pre- and postprocessing yourself, here's how to do that:

```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AutoModel
>>> import torch

>>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224")
>>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> candidate_labels = ["2 cats", "2 dogs"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.'


key: _node_content
string_value:
{"id_": "46c8aec0-1503-4f90-8b7c-f4e993986f16", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ef21361-6443-413f-a86e-5d9bda19ad5d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "a479b257c24371ddf1d442cedaddf6ee6bed2acf1cf9a3a86e11992be5c6d4a5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4306c05c-a5bb-4668-8d4c-af45d9335822", "node_type": "1", "metadata": {}, "hash": "f0ff1f7a0d2c2b1df80b7422161ad67e3a91f0a1ed985764b0a01aed5b780737", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2733, "end_char_idx": 5305, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Pipeline API

The pipeline allows to use the model in a few lines of code:

```python
>>> from transformers import pipeline
>>> from PIL import Image
>>> import requests

>>> # load pipe
>>> image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-base-patch16-224")

>>> # load image
>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> # inference
>>> candidate_labels = ["2 cats", "a plane", "a remote"]
>>> outputs = image_classifier(image, candidate_labels=candidate_labels)
>>> outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
>>> print(outputs)
[{'score': 0.1979, 'label': '2 cats'}, {'score': 0.0, 'label': 'a remote'}, {'score': 0.0, 'label': 'a plane'}]
```

### Using the model yourself

If you want to do the pre- and postprocessing yourself, here's how to do that:

```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, AutoModel
>>> import torch

>>> model = AutoModel.from_pretrained("google/siglip-base-patch16-224")
>>> processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> candidate_labels = ["2 cats", "2 dogs"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]
>>> # important: we pass `padding=max_length` since the model was trained with this
>>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits_per_image = outputs.logits_per_image
>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities
>>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'")
31.9% that image 0 is 'a photo of 2 cats'
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SigLIP.

- [Zero-shot image classification task guide](../tasks/zero_shot_image_classification_md)
- Demo notebooks for SigLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SigLIP). 🌎

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "4306c05c-a5bb-4668-8d4c-af45d9335822", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46c8aec0-1503-4f90-8b7c-f4e993986f16", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "0bb46e7113b19679fbda4e938414fb37e1f2deb469efd0e708e277cc962f6e80", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "927ebaf6-4d71-47a0-a1fb-058f4d876f3d", "node_type": "1", "metadata": {}, "hash": "3edbd5c5dd12dd084626cea7668416fc55fb582760d2106086ede213cd3aede1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4240, "end_char_idx": 7141, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
for label in candidate_labels]
>>> # important: we pass `padding=max_length` since the model was trained with this
>>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> logits_per_image = outputs.logits_per_image
>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities
>>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'")
31.9% that image 0 is 'a photo of 2 cats'
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SigLIP.

- [Zero-shot image classification task guide](../tasks/zero_shot_image_classification_md)
- Demo notebooks for SigLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SigLIP). 🌎

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


## Combining SigLIP and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> import requests
>>> from PIL import Image
>>> from transformers import SiglipProcessor, SiglipModel
>>> device = "cuda" # the device to load the model onto

>>> model = SiglipModel.from_pretrained(
...     "google/siglip-so400m-patch14-384",
...     attn_implementation="flash_attention_2",
...     torch_dtype=torch.float16,
...     device_map=device,
... )
>>> processor = SiglipProcessor.from_pretrained("google/siglip-so400m-patch14-384")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> candidate_labels = ["2 cats", "2 dogs"]
# follows the pipeline prompt template to get same results
>>> candidate_labels = [f'This is a photo of {label}.' for label in candidate_labels]
# important: we pass `padding=max_length` since the model was trained with this
>>> inputs = processor(text=candidate_labels, images=image, padding="max_length", return_tensors="pt")
>>> inputs.to(device)

>>> with torch.no_grad():
...     with torch.autocast(device):
...         outputs = model(**inputs)

>>> logits_per_image = outputs.logits_per_image
>>> probs = torch.sigmoid(logits_per_image) # these are the probabilities
>>> print(f"{probs[0][0]:.1%} that image 0 is '{candidate_labels[0]}'")
51.3% that image 0 is 'This is a photo of 2 cats.'
```


key: _node_content
string_value:
{"id_": "927ebaf6-4d71-47a0-a1fb-058f4d876f3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deacb9c7-75ab-464d-adff-6f38f7931701", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "c5e211219231a89121d9f8f1930d41d4def76c363b0aba344eb20d2975703e02", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4306c05c-a5bb-4668-8d4c-af45d9335822", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/siglip", "title": "SigLIP", "tokens": 2478, "retrieve_doc": true, "source": "transformers"}, "hash": "0d6885281b80b90be9fabf8b5bbac91e30556baa2e844a7710d01d669c18c5b8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7144, "end_char_idx": 9400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

You may set `attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used. Make sure you have `torch>=2.1.1`.

```python
>>> from transformers import SiglipModel

>>> model = SiglipModel.from_pretrained(
...     "google/siglip-so400m-patch14-384",
...     attn_implementation="sdpa",
...     torch_dtype=torch.float16,
...     device_map=device,
... )
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).


## Expected speedups

Below is an expected speedup diagram that compares inference time between the native implementation in transformers using `google/siglip-so400m-patch14-384` checkpoint in `float16` precision and the Flash Attention 2 / SDPA version of the model using different batch sizes.

<div style="text-align: center">
<img src="https://i.imgur.com/cWm4rsn.png">
</div>


## SiglipConfig

[[autodoc]] SiglipConfig
    - from_text_vision_configs

## SiglipTextConfig

[[autodoc]] SiglipTextConfig

## SiglipVisionConfig

[[autodoc]] SiglipVisionConfig

## SiglipTokenizer

[[autodoc]] SiglipTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## SiglipImageProcessor

[[autodoc]] SiglipImageProcessor
    - preprocess

## SiglipProcessor

[[autodoc]] SiglipProcessor

## SiglipModel

[[autodoc]] SiglipModel
    - forward
    - get_text_features
    - get_image_features

## SiglipTextModel

[[autodoc]] SiglipTextModel
    - forward

## SiglipVisionModel

[[autodoc]] SiglipVisionModel
    - forward


## SiglipForImageClassification

[[autodoc]] SiglipForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "da911c9c-785d-474a-8275-aea72417da22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers"}, "hash": "b2017e88fe0136f82e097e15171a99d7f9987bc6c4222552a59a4da36795544d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1e31b402-b7eb-4517-9b60-c0e6e20e1dfb", "node_type": "1", "metadata": {}, "hash": "b248ac5d29fed31524e38895ab8ecff7c23812b40791f4125c134ad413f249cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3357, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PLBart

## Overview

The PLBART model was proposed in [Unified Pre-training for Program Understanding and Generation](https://arxiv.org/abs/2103.06333) by Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang.
This is a BART-like model which can be used to perform code-summarization, code-generation, and code-translation tasks. The pre-trained model `plbart-base` has been trained using multilingual denoising task
on Java, Python and English.

According to the abstract

*Code summarization and generation empower conversion between programming language (PL) and natural language (NL),
while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, 
a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks.
PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding.
Experiments on code summarization in the English language, code generation, and code translation in seven programming languages
show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program
repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding.
Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow
(e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels
even with limited annotations.*

This model was contributed by [gchhablani](https://huggingface.co/gchhablani). The Authors' code can be found [here](https://github.com/wasiahmad/PLBART).

## Usage examples

PLBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for code-to-text, text-to-code, code-to-code tasks. As the
model is multilingual it expects the sequences in a different format. A special language id token is added in both the
source and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The
target text format is `[tgt_lang_code] X [eos]`. `bos` is never used.

However, for fine-tuning, in some cases no language token is provided in cases where a single language is used. Please refer to [the paper](https://arxiv.org/abs/2103.06333) to learn more about this.

In cases where the language code is needed, the regular [`~PLBartTokenizer.__call__`] will encode source text format 
when you pass texts as the first argument or with the keyword argument `text`, and will encode target text format if
it's passed with the `text_target` keyword argument.

### Supervised training

```python
>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer

>>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-base", src_lang="en_XX", tgt_lang="python")
>>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])"
>>> expected_translation_english = "Returns the maximum value of a b c."
>>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt")
>>> model(**inputs)
```

### Generation

  While generating the target text set the `decoder_start_token_id` to the target language id.


key: _node_content
string_value:
{"id_": "1e31b402-b7eb-4517-9b60-c0e6e20e1dfb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59cf7eb1-f6b6-4ca8-bad3-c4dac6bdf6dd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers"}, "hash": "b2017e88fe0136f82e097e15171a99d7f9987bc6c4222552a59a4da36795544d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da911c9c-785d-474a-8275-aea72417da22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/plbart", "title": "PLBart", "tokens": 1313, "retrieve_doc": true, "source": "transformers"}, "hash": "60c88eaaf2404eb53eb93210f3cc90abc5851d84628bb88d7fbe71116090800c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1779, "end_char_idx": 4835, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage examples

PLBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for code-to-text, text-to-code, code-to-code tasks. As the
model is multilingual it expects the sequences in a different format. A special language id token is added in both the
source and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The
target text format is `[tgt_lang_code] X [eos]`. `bos` is never used.

However, for fine-tuning, in some cases no language token is provided in cases where a single language is used. Please refer to [the paper](https://arxiv.org/abs/2103.06333) to learn more about this.

In cases where the language code is needed, the regular [`~PLBartTokenizer.__call__`] will encode source text format 
when you pass texts as the first argument or with the keyword argument `text`, and will encode target text format if
it's passed with the `text_target` keyword argument.

### Supervised training

```python
>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer

>>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-base", src_lang="en_XX", tgt_lang="python")
>>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])"
>>> expected_translation_english = "Returns the maximum value of a b c."
>>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt")
>>> model(**inputs)
```

### Generation

  While generating the target text set the `decoder_start_token_id` to the target language id. The following
  example shows how to translate Python to English using the `uclanlp/plbart-python-en_XX` model.

```python
>>> from transformers import PLBartForConditionalGeneration, PLBartTokenizer

>>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-python-en_XX", src_lang="python", tgt_lang="en_XX")
>>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])"
>>> inputs = tokenizer(example_python_phrase, return_tensors="pt")
>>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-python-en_XX")
>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id["en_XX"])
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"Returns the maximum value of a b c."
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## PLBartConfig

[[autodoc]] PLBartConfig

## PLBartTokenizer

[[autodoc]] PLBartTokenizer
    - build_inputs_with_special_tokens

## PLBartModel

[[autodoc]] PLBartModel
    - forward

## PLBartForConditionalGeneration

[[autodoc]] PLBartForConditionalGeneration
    - forward

## PLBartForSequenceClassification

[[autodoc]] PLBartForSequenceClassification
    - forward

## PLBartForCausalLM

[[autodoc]] PLBartForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "5ebe8848-4094-444a-84dd-d67fa052aedb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "501305bc-fcf7-4429-86e4-3d1e8c07dc20", "node_type": "1", "metadata": {}, "hash": "f68922bd49394cfb3f1e7866e006558355b27bedf0eb11e95c4a9d3a9e21ab15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3290, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# T5

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=t5">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-t5-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/t5-base">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
<a href="https://huggingface.co/papers/1910.10683">
<img alt="Paper page" src="https://img.shields.io/badge/Paper%20page-1910.10683-green">
</a>
</div>

## Overview

The T5 model was presented in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) by [Colin Raffel](https://huggingface.co/craffel), Noam Shazeer, [Adam Roberts](https://huggingface.co/adarob), Katherine Lee, Sharan Narang,
Michael Matena, Yanqi Zhou, Wei Li, [Peter J. Liu](https://huggingface.co/peterjliu).

The abstract from the paper is the following:

*Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream
task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning
has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of
transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a
text-to-text format. Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer
approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration
with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering
summarization, question answering, text classification, and more. To facilitate future work on transfer learning for
NLP, we release our dataset, pre-trained models, and code.*

All checkpoints can be found on the [hub](https://huggingface.co/models?search=t5).

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).

## Usage tips

- T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which
each task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a
different prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,
for summarization: *summarize: ...*.
- The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).
- Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens.


key: _node_content
string_value:
{"id_": "501305bc-fcf7-4429-86e4-3d1e8c07dc20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ebe8848-4094-444a-84dd-d67fa052aedb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "9fbdcb72acc43d1bec68a88d3dda44b8cc2548e81b864558560ac3ba23399ab0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba0eeb09-a79c-4f22-aa22-a3f5db811116", "node_type": "1", "metadata": {}, "hash": "52522f4920ba72cef3a57155961d36f56a76c52177fd6d68f05ba5a22a66927d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1426, "end_char_idx": 4545, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Our systematic study compares pretraining objectives, architectures, unlabeled datasets, transfer
approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration
with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering
summarization, question answering, text classification, and more. To facilitate future work on transfer learning for
NLP, we release our dataset, pre-trained models, and code.*

All checkpoints can be found on the [hub](https://huggingface.co/models?search=t5).

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/text-to-text-transfer-transformer).

## Usage tips

- T5 is an encoder-decoder model pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which
each task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a
different prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,
for summarization: *summarize: ...*.
- The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).
- Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens.

- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.

- See the [training](#training), [inference](#inference) and [resources](#resources) sections below for all details regarding usage.

T5 comes in different sizes:

- [google-t5/t5-small](https://huggingface.co/google-t5/t5-small)

- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base)

- [google-t5/t5-large](https://huggingface.co/google-t5/t5-large)

- [google-t5/t5-3b](https://huggingface.co/google-t5/t5-3b)

- [google-t5/t5-11b](https://huggingface.co/google-t5/t5-11b).

Based on the original T5 model, Google has released some follow-up works:

- **T5v1.1**: T5v1.1 is an improved version of T5 with some architectural tweaks, and is pre-trained on C4 only without
  mixing in the supervised tasks. Refer to the documentation of T5v1.1 which can be found [here](t5v1.1).

- **mT5**: mT5 is a multilingual T5 model. It is pre-trained on the mC4 corpus, which includes 101 languages. Refer to
  the documentation of mT5 which can be found [here](mt5).

- **byT5**: byT5 is a T5 model pre-trained on byte sequences rather than SentencePiece subword token sequences. Refer
  to the documentation of byT5 which can be found [here](byt5).


key: _node_content
string_value:
{"id_": "ba0eeb09-a79c-4f22-aa22-a3f5db811116", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "501305bc-fcf7-4429-86e4-3d1e8c07dc20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "3846021b12366a62c285cee3e21ea0d6b6fdcab1c820793363b6dc450b2df969", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cfdc9111-af57-49d3-98c8-ddeb5b325a56", "node_type": "1", "metadata": {}, "hash": "54c4a5bfc4c3ccebb41f3ad9d8bb7771056809f593ba47b7258e46411c9bff66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3292, "end_char_idx": 6094, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.

- See the [training](#training), [inference](#inference) and [resources](#resources) sections below for all details regarding usage.

T5 comes in different sizes:

- [google-t5/t5-small](https://huggingface.co/google-t5/t5-small)

- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base)

- [google-t5/t5-large](https://huggingface.co/google-t5/t5-large)

- [google-t5/t5-3b](https://huggingface.co/google-t5/t5-3b)

- [google-t5/t5-11b](https://huggingface.co/google-t5/t5-11b).

Based on the original T5 model, Google has released some follow-up works:

- **T5v1.1**: T5v1.1 is an improved version of T5 with some architectural tweaks, and is pre-trained on C4 only without
  mixing in the supervised tasks. Refer to the documentation of T5v1.1 which can be found [here](t5v1.1).

- **mT5**: mT5 is a multilingual T5 model. It is pre-trained on the mC4 corpus, which includes 101 languages. Refer to
  the documentation of mT5 which can be found [here](mt5).

- **byT5**: byT5 is a T5 model pre-trained on byte sequences rather than SentencePiece subword token sequences. Refer
  to the documentation of byT5 which can be found [here](byt5).

- **UL2**: UL2 is a T5 like model pretrained on various denoising objectives

- **Flan-T5**: Flan is a pretraining methods that is based on prompting. The Flan-T5 are T5 models trained on the Flan collection of 
    datasets which include: `taskmaster2`, `djaym7/wiki_dialog`, `deepmind/code_contests`, `lambada`, `gsm8k`, `aqua_rat`, `esnli`, `quasc` and `qed`.

- **FLan-UL2** : the UL2 model finetuned using the "Flan" prompt tuning and dataset collection.

- **UMT5**: UmT5 is a multilingual T5 model trained on an improved and refreshed mC4 multilingual corpus,  29 trillion characters across 107 language, using a new sampling method, UniMax. Refer to
 the documentation of mT5 which can be found [here](umt5).

## Training

T5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher
forcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input
sequence is fed to the model using `input_ids`. The target sequence is shifted to the right, i.e., prepended by a
start-sequence token and fed to the decoder using the `decoder_input_ids`. In teacher-forcing style, the target
sequence is then appended by the EOS token and corresponds to the `labels`. The PAD token is hereby used as the
start-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.

One can use [`T5ForConditionalGeneration`] (or the Tensorflow/Flax variant), which includes the
language modeling head on top of the decoder.


key: _node_content
string_value:
{"id_": "cfdc9111-af57-49d3-98c8-ddeb5b325a56", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba0eeb09-a79c-4f22-aa22-a3f5db811116", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "21381c074555690cf11e8a1785dbc8de89fdcc2b4abe0820584b64db48a607d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1b1de9d0-5e2d-4769-a4cf-f4e6066a258b", "node_type": "1", "metadata": {}, "hash": "bb633e13c4f3b9c1b17eecb5489dcd6ff77b3c3b9b0d9403e8a375645edbd533", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4698, "end_char_idx": 7660, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Flan-T5 are T5 models trained on the Flan collection of 
    datasets which include: `taskmaster2`, `djaym7/wiki_dialog`, `deepmind/code_contests`, `lambada`, `gsm8k`, `aqua_rat`, `esnli`, `quasc` and `qed`.

- **FLan-UL2** : the UL2 model finetuned using the "Flan" prompt tuning and dataset collection.

- **UMT5**: UmT5 is a multilingual T5 model trained on an improved and refreshed mC4 multilingual corpus,  29 trillion characters across 107 language, using a new sampling method, UniMax. Refer to
 the documentation of mT5 which can be found [here](umt5).

## Training

T5 is an encoder-decoder model and converts all NLP problems into a text-to-text format. It is trained using teacher
forcing. This means that for training, we always need an input sequence and a corresponding target sequence. The input
sequence is fed to the model using `input_ids`. The target sequence is shifted to the right, i.e., prepended by a
start-sequence token and fed to the decoder using the `decoder_input_ids`. In teacher-forcing style, the target
sequence is then appended by the EOS token and corresponds to the `labels`. The PAD token is hereby used as the
start-sequence token. T5 can be trained / fine-tuned both in a supervised and unsupervised fashion.

One can use [`T5ForConditionalGeneration`] (or the Tensorflow/Flax variant), which includes the
language modeling head on top of the decoder.

- Unsupervised denoising training

In this setup, spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and
the output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each
sentinel token represents a unique mask token for this sentence and should start with `<extra_id_0>`,
`<extra_id_1>`, ... up to `<extra_id_99>`. As a default, 100 sentinel tokens are available in
[`T5Tokenizer`].

For instance, the sentence "The cute dog walks in the park" with the masks put on "cute dog" and "the" should be
processed as follows:

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids
>>> labels = tokenizer("<extra_id_0> cute dog <extra_id_1> the <extra_id_2>", return_tensors="pt").input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
>>> loss.item()
3.7837
```

If you're interested in pre-training T5 on a new corpus, check out the [run_t5_mlm_flax.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) script in the Examples
directory.

- Supervised training

In this setup, the input sequence and output sequence are a standard sequence-to-sequence input-output mapping.


key: _node_content
string_value:
{"id_": "1b1de9d0-5e2d-4769-a4cf-f4e6066a258b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cfdc9111-af57-49d3-98c8-ddeb5b325a56", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "ce295dc9dfd08ef08f85f5cdfa8b992c81c4f730ca9cf33d55625b28cf3fa0ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c9eb92c0-5e5a-4bfe-861f-4267249604cf", "node_type": "1", "metadata": {}, "hash": "a8a16971c0d7f35c23d0d23f4ec6274421fe3c2d777b62ef0e1559313e9a40b5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6096, "end_char_idx": 9299, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Unsupervised denoising training

In this setup, spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and
the output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each
sentinel token represents a unique mask token for this sentence and should start with `<extra_id_0>`,
`<extra_id_1>`, ... up to `<extra_id_99>`. As a default, 100 sentinel tokens are available in
[`T5Tokenizer`].

For instance, the sentence "The cute dog walks in the park" with the masks put on "cute dog" and "the" should be
processed as follows:

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids
>>> labels = tokenizer("<extra_id_0> cute dog <extra_id_1> the <extra_id_2>", return_tensors="pt").input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
>>> loss.item()
3.7837
```

If you're interested in pre-training T5 on a new corpus, check out the [run_t5_mlm_flax.py](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) script in the Examples
directory.

- Supervised training

In this setup, the input sequence and output sequence are a standard sequence-to-sequence input-output mapping.
Suppose that we want to fine-tune the model for translation for example, and we have a training example: the input
sequence "The house is wonderful." and output sequence "Das Haus ist wunderbar.", then they should be prepared for
the model as follows:

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> input_ids = tokenizer("translate English to German: The house is wonderful.", return_tensors="pt").input_ids
>>> labels = tokenizer("Das Haus ist wunderbar.", return_tensors="pt").input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
>>> loss.item()
0.2542
```

As you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the
`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded
target sequence). The model will automatically create the `decoder_input_ids` based on the `labels`, by
shifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is
equal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate
English to German: ' before encoding it. This will help in improving the performance, as this task prefix was used
during T5's pre-training.

However, the example above only shows a single training example. In practice, one trains deep learning models in
batches.


key: _node_content
string_value:
{"id_": "c9eb92c0-5e5a-4bfe-861f-4267249604cf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1b1de9d0-5e2d-4769-a4cf-f4e6066a258b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "15fab9815d1ab9f894595f1ba42cc0f038638e3bba37c025c34942a1c9c60f89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3fd70fc2-d5dd-4931-b015-f9309ffc9584", "node_type": "1", "metadata": {}, "hash": "e6b69ae207503c9dc0da54c984ced6591351883a4949717d29a4acfeb09bea86", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7661, "end_char_idx": 10278, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Suppose that we want to fine-tune the model for translation for example, and we have a training example: the input
sequence "The house is wonderful." and output sequence "Das Haus ist wunderbar.", then they should be prepared for
the model as follows:

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> input_ids = tokenizer("translate English to German: The house is wonderful.", return_tensors="pt").input_ids
>>> labels = tokenizer("Das Haus ist wunderbar.", return_tensors="pt").input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_ids=input_ids, labels=labels).loss
>>> loss.item()
0.2542
```

As you can see, only 2 inputs are required for the model in order to compute a loss: `input_ids` (which are the
`input_ids` of the encoded input sequence) and `labels` (which are the `input_ids` of the encoded
target sequence). The model will automatically create the `decoder_input_ids` based on the `labels`, by
shifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is
equal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate
English to German: ' before encoding it. This will help in improving the performance, as this task prefix was used
during T5's pre-training.

However, the example above only shows a single training example. In practice, one trains deep learning models in
batches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one
typically defines a `max_source_length` and `max_target_length`, which determine the maximum length of the
input and output sequences respectively (otherwise they are truncated). These should be carefully set depending on
the task.

In addition, we must make sure that padding token id's of the `labels` are not taken into account by the loss
function. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the `ignore_index`
of the `CrossEntropyLoss`. In Flax, one can use the `decoder_attention_mask` to ignore padded tokens from
the loss (see the [Flax summarization script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization) for details). We also pass
`attention_mask` as additional input to the model, which makes sure that padding tokens of the inputs are
ignored. The code example below illustrates all of this.


key: _node_content
string_value:
{"id_": "3fd70fc2-d5dd-4931-b015-f9309ffc9584", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c9eb92c0-5e5a-4bfe-861f-4267249604cf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "a9c0fcf18b2caac0ea1892e5c9870c6e88e5d912ca4b0f51e3211149bc171e71", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "582bc3c4-c231-43c1-a404-0eccb803b3db", "node_type": "1", "metadata": {}, "hash": "bd35e6532e00f9a90d7557771087ff9ccfef3215765ffad0a31f4965c238966d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8726, "end_char_idx": 12002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The model will automatically create the `decoder_input_ids` based on the `labels`, by
shifting them one position to the right and prepending the `config.decoder_start_token_id`, which for T5 is
equal to 0 (i.e. the id of the pad token). Also note the task prefix: we prepend the input sequence with 'translate
English to German: ' before encoding it. This will help in improving the performance, as this task prefix was used
during T5's pre-training.

However, the example above only shows a single training example. In practice, one trains deep learning models in
batches. This entails that we must pad/truncate examples to the same length. For encoder-decoder models, one
typically defines a `max_source_length` and `max_target_length`, which determine the maximum length of the
input and output sequences respectively (otherwise they are truncated). These should be carefully set depending on
the task.

In addition, we must make sure that padding token id's of the `labels` are not taken into account by the loss
function. In PyTorch and Tensorflow, this can be done by replacing them with -100, which is the `ignore_index`
of the `CrossEntropyLoss`. In Flax, one can use the `decoder_attention_mask` to ignore padded tokens from
the loss (see the [Flax summarization script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization) for details). We also pass
`attention_mask` as additional input to the model, which makes sure that padding tokens of the inputs are
ignored. The code example below illustrates all of this.

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration
>>> import torch

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> # the following 2 hyperparameters are task-specific
>>> max_source_length = 512
>>> max_target_length = 128

>>> # Suppose we have the following 2 training examples:
>>> input_sequence_1 = "Welcome to NYC"
>>> output_sequence_1 = "Bienvenue à NYC"

>>> input_sequence_2 = "HuggingFace is a company"
>>> output_sequence_2 = "HuggingFace est une entreprise"

>>> # encode the inputs
>>> task_prefix = "translate English to French: "
>>> input_sequences = [input_sequence_1, input_sequence_2]

>>> encoding = tokenizer(
...     [task_prefix + sequence for sequence in input_sequences],
...     padding="longest",
...     max_length=max_source_length,
...     truncation=True,
...     return_tensors="pt",
... )

>>> input_ids, attention_mask = encoding.input_ids, encoding.attention_mask

>>> # encode the targets
>>> target_encoding = tokenizer(
...     [output_sequence_1, output_sequence_2],
...     padding="longest",
...     max_length=max_target_length,
...     truncation=True,
...     return_tensors="pt",
... )
>>> labels = target_encoding.input_ids

>>> # replace padding token id's of the labels by -100 so it's ignored by the loss
>>> labels[labels == tokenizer.pad_token_id] = -100

>>> # forward pass
>>> loss = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels).loss
>>> loss.item()
0.188
```

Additional training tips:

- T5 models need a slightly higher learning rate than the default one set in the `Trainer` when using the AdamW
optimizer.


key: _node_content
string_value:
{"id_": "582bc3c4-c231-43c1-a404-0eccb803b3db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3fd70fc2-d5dd-4931-b015-f9309ffc9584", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "318027b0eacb926bf790535c00ae848d0aacc9927eda43669d7d69f6b35244b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23eb8394-e627-4cef-9f6b-a4dae5d7484b", "node_type": "1", "metadata": {}, "hash": "f8b8aca254aa0e29bab97502b7aa35174c9b86ccb4aff1c7038476cf1c460d4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12003, "end_char_idx": 15403, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Typically, 1e-4 and 3e-4 work well for most problems (classification, summarization, translation, question
answering, question generation). Note that T5 was pre-trained using the AdaFactor optimizer.

According to [this forum post](https://discuss.huggingface.co/t/t5-finetuning-tips/684), task prefixes matter when
(1) doing multi-task training (2) your task is similar or related to one of the supervised tasks used in T5's
pre-training mixture (see Appendix D of the [paper](https://arxiv.org/pdf/1910.10683.pdf) for the task prefixes
used).

If training on TPU, it is recommended to pad all examples of the dataset to the same length or make use of
*pad_to_multiple_of* to have a small number of predefined bucket sizes to fit all examples in. Dynamically padding
batches to the longest example is not recommended on TPU as it triggers a recompilation for every batch shape that is
encountered during training thus significantly slowing down the training. only padding up to the longest example in a
batch) leads to very slow training on TPU.

## Inference

At inference time, it is recommended to use [`~generation.GenerationMixin.generate`]. This
method takes care of encoding the input and feeding the encoded hidden states via cross-attention layers to the decoder
and auto-regressively generates the decoder output. Check out [this blog post](https://huggingface.co/blog/how-to-generate) to know all the details about generating text with Transformers.
There's also [this blog post](https://huggingface.co/blog/encoder-decoder#encoder-decoder) which explains how
generation works in general in encoder-decoder models.

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> input_ids = tokenizer("translate English to German: The house is wonderful.", return_tensors="pt").input_ids
>>> outputs = model.generate(input_ids)
>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Das Haus ist wunderbar.
```

Note that T5 uses the `pad_token_id` as the `decoder_start_token_id`, so when doing generation without using
[`~generation.GenerationMixin.generate`], make sure you start it with the `pad_token_id`.

The example above only shows a single example. You can also do batched inference, like so:

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> task_prefix = "translate English to German: "
>>> # use different length sentences to test batching
>>> sentences = ["The house is wonderful.", "I like to work in NYC."]

>>> inputs = tokenizer([task_prefix + sentence for sentence in sentences], return_tensors="pt", padding=True)

>>> output_sequences = model.generate(
...     input_ids=inputs["input_ids"],
...     attention_mask=inputs["attention_mask"],
...     do_sample=False,  # disable sampling to test if batching affects output
... )

>>> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))
['Das Haus ist wunderbar.', 'Ich arbeite gerne in NYC.']
```

Because T5 has been trained with the span-mask denoising objective,
it can be used to predict the sentinel (masked-out) tokens during inference.


key: _node_content
string_value:
{"id_": "23eb8394-e627-4cef-9f6b-a4dae5d7484b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "582bc3c4-c231-43c1-a404-0eccb803b3db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "a9bcb5e14a8a2f5ccbe9696883824a5e157e3a9f3eb16260c36b5d91bb9a51fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5fe22a1-52e6-46af-a8e1-6359dd20b96a", "node_type": "1", "metadata": {}, "hash": "aaadb9ca0ec2d5b4fa1e4c683e430ca143106d255ebdf553d31d09019a2b6a53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13934, "end_char_idx": 17180, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", return_tensors="pt").input_ids
>>> outputs = model.generate(input_ids)
>>> print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Das Haus ist wunderbar.
```

Note that T5 uses the `pad_token_id` as the `decoder_start_token_id`, so when doing generation without using
[`~generation.GenerationMixin.generate`], make sure you start it with the `pad_token_id`.

The example above only shows a single example. You can also do batched inference, like so:

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> task_prefix = "translate English to German: "
>>> # use different length sentences to test batching
>>> sentences = ["The house is wonderful.", "I like to work in NYC."]

>>> inputs = tokenizer([task_prefix + sentence for sentence in sentences], return_tensors="pt", padding=True)

>>> output_sequences = model.generate(
...     input_ids=inputs["input_ids"],
...     attention_mask=inputs["attention_mask"],
...     do_sample=False,  # disable sampling to test if batching affects output
... )

>>> print(tokenizer.batch_decode(output_sequences, skip_special_tokens=True))
['Das Haus ist wunderbar.', 'Ich arbeite gerne in NYC.']
```

Because T5 has been trained with the span-mask denoising objective,
it can be used to predict the sentinel (masked-out) tokens during inference.
The predicted tokens will then be placed between the sentinel tokens.

```python
>>> from transformers import T5Tokenizer, T5ForConditionalGeneration

>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-small")
>>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small")

>>> input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids

>>> sequence_ids = model.generate(input_ids)
>>> sequences = tokenizer.batch_decode(sequence_ids)
>>> sequences
['<pad> <extra_id_0> park offers <extra_id_1> the <extra_id_2> park.</s>']
```

## Performance

If you'd like a faster training and inference performance, install [NVIDIA APEX](https://github.com/NVIDIA/apex#quick-start) for NVIDIA GPUs, or [ROCm APEX](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs and then the model will automatically use `apex.normalization.FusedRMSNorm` instead of `T5LayerNorm`. The former uses an optimized fused kernel which is several times faster than the latter.


## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with T5. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A notebook for how to [finetune T5 for classification and multiple choice](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).
- A notebook for how to [finetune T5 for sentiment span extraction](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb).


key: _node_content
string_value:
{"id_": "e5fe22a1-52e6-46af-a8e1-6359dd20b96a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23eb8394-e627-4cef-9f6b-a4dae5d7484b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "e611367013c76f92267a4b6d9c51677bcf242c2fcd9f20bc7a549ecde5437e6f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb", "node_type": "1", "metadata": {}, "hash": "e0846fb07b468242044b588834cf5575d8e8886a8a335e2b73ce5b974ccfd3cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16330, "end_char_idx": 19274, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The former uses an optimized fused kernel which is several times faster than the latter.


## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with T5. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A notebook for how to [finetune T5 for classification and multiple choice](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).
- A notebook for how to [finetune T5 for sentiment span extraction](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb). 🌎

<PipelineTag pipeline="token-classification"/>

- A notebook for how to [finetune T5 for named entity recognition](https://colab.research.google.com/drive/1obr78FY_cBmWY5ODViCmzdY6O1KB65Vc?usp=sharing). 🌎

<PipelineTag pipeline="text-generation"/>

- A notebook for [Finetuning CodeT5 for generating docstrings from Ruby code](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/T5/Fine_tune_CodeT5_for_generating_docstrings_from_Ruby_code.ipynb).

<PipelineTag pipeline="summarization"/>

- A notebook to [Finetune T5-base-dutch to perform Dutch abstractive summarization on a TPU](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/T5/Fine_tuning_Dutch_T5_base_on_CNN_Daily_Mail_for_summarization_(on_TPU_using_HuggingFace_Accelerate).ipynb).
- A notebook for how to [finetune T5 for summarization in PyTorch and track experiments with WandB](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb#scrollTo=OKRpFvYhBauC). 🌎
- A blog post on [Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).
- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).
- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).
- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).
- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the 🤗 Hugging Face course.


key: _node_content
string_value:
{"id_": "f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5fe22a1-52e6-46af-a8e1-6359dd20b96a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "b7af59cb6076a6d4fad1d83f0bd6b43e14266390bba22b174d67d68287a5408e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "717adf7a-1ca2-4e96-a3b5-4a1dfa69fb33", "node_type": "1", "metadata": {}, "hash": "7aae1798735c66dba75a40823f3a1a053805c1176ae65f33a962872f835a67b5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18000, "end_char_idx": 20960, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- A notebook for how to [finetune T5 for summarization in PyTorch and track experiments with WandB](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb#scrollTo=OKRpFvYhBauC). 🌎
- A blog post on [Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq).
- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization.ipynb).
- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/summarization-tf.ipynb).
- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/summarization).
- [Summarization](https://huggingface.co/course/chapter7/5?fw=pt#summarization) chapter of the 🤗 Hugging Face course.
- [Summarization task guide](../tasks/summarization)

<PipelineTag pipeline="fill-mask"/>

- [`FlaxT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#t5-like-span-masked-language-modeling) for training T5 with a span-masked language model objective. The script also shows how to train a T5 tokenizer. [`FlaxT5ForConditionalGeneration`] is also supported by this [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).

<PipelineTag pipeline="translation"/>

- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).
- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).
- [Translation task guide](../tasks/translation)

<PipelineTag pipeline="question-answering"/>

- A notebook on how to [finetune T5 for question answering with TensorFlow 2](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb). 🌎
- A notebook on how to [finetune T5 for question answering on a TPU](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil).


key: _node_content
string_value:
{"id_": "717adf7a-1ca2-4e96-a3b5-4a1dfa69fb33", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1c3fd25-ee81-4007-ade4-89527a28caf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "5e460bba7e0b0df3f8c895dffadb82a9921a5753e34a77b5d31c10667662565d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0abd6ae-9a70-49bb-9811-1aaab8e8f6fb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/t5", "title": "T5", "tokens": 5728, "retrieve_doc": true, "source": "transformers"}, "hash": "41e6f7d959dc03f25232c1b699e700a631b1c90760839cc88043e83f6d8d7740", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19627, "end_char_idx": 22332, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The script also shows how to train a T5 tokenizer. [`FlaxT5ForConditionalGeneration`] is also supported by this [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).

<PipelineTag pipeline="translation"/>

- [`T5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb).
- [`TFT5ForConditionalGeneration`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/translation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).
- [Translation task guide](../tasks/translation)

<PipelineTag pipeline="question-answering"/>

- A notebook on how to [finetune T5 for question answering with TensorFlow 2](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb). 🌎
- A notebook on how to [finetune T5 for question answering on a TPU](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil).

🚀 **Deploy**
- A blog post on how to deploy [T5 11B for inference for less than $500](https://www.philschmid.de/deploy-t5-11b).

## T5Config

[[autodoc]] T5Config

## T5Tokenizer

[[autodoc]] T5Tokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## T5TokenizerFast

[[autodoc]] T5TokenizerFast

<frameworkcontent>
<pt>

## T5Model

[[autodoc]] T5Model
    - forward

## T5ForConditionalGeneration

[[autodoc]] T5ForConditionalGeneration
    - forward

## T5EncoderModel

[[autodoc]] T5EncoderModel
    - forward

## T5ForSequenceClassification

[[autodoc]] T5ForSequenceClassification
    - forward

## T5ForTokenClassification

[[autodoc]] T5ForTokenClassification
    - forward

## T5ForQuestionAnswering

[[autodoc]] T5ForQuestionAnswering
    - forward

</pt>
<tf>

## TFT5Model

[[autodoc]] TFT5Model
    - call

## TFT5ForConditionalGeneration

[[autodoc]] TFT5ForConditionalGeneration
    - call

## TFT5EncoderModel

[[autodoc]] TFT5EncoderModel
    - call

</tf>
<jax>

## FlaxT5Model

[[autodoc]] FlaxT5Model
    - __call__
    - encode
    - decode

## FlaxT5ForConditionalGeneration

[[autodoc]] FlaxT5ForConditionalGeneration
    - __call__
    - encode
    - decode

## FlaxT5EncoderModel

[[autodoc]] FlaxT5EncoderModel
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "f11fd7dd-6584-4f69-bd95-5a794ee5121d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23c21ffd-fff2-4ac5-b292-68fd917442c7", "node_type": "1", "metadata": {}, "hash": "5c546cc2f33c219d055b079dd32ff47547c5ee34f803231abcd7426b847fcc0e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2776, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OpenAI GPT2

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=gpt2">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-gpt2-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/gpt2">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

OpenAI GPT-2 model was proposed in [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec
Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever from [OpenAI](https://huggingface.co/openai). It's a causal (unidirectional)
transformer pretrained using language modeling on a very large corpus of ~40 GB of text data.

The abstract from the paper is the following:

*GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million
web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some
text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks
across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than
10X the amount of data.*

[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by
Hugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five
different sizes: small, medium, large, xl and a distilled version of the small checkpoint: *distilgpt-2*.

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://openai.com/blog/better-language-models/).

## Usage tips

- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be
  observed in the *run_generation.py* example script.
- The model can take the *past_key_values* (for PyTorch) or *past* (for TF) as input, which is the previously computed
  key/value attention pairs. Using this (*past_key_values* or *past*) value prevents the model from re-computing
  pre-computed values in the context of text generation. For PyTorch, see *past_key_values* argument of the
  [`GPT2Model.forward`] method, or for TF the *past* argument of the
  [`TFGPT2Model.call`] method for more information on its usage.


key: _node_content
string_value:
{"id_": "23c21ffd-fff2-4ac5-b292-68fd917442c7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f11fd7dd-6584-4f69-bd95-5a794ee5121d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "2afcc1e6695848523a447a8fd4aae9c493a3019851468330c043eb0fe0ff51a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5646deeb-9fae-4061-955e-8c8853e558b0", "node_type": "1", "metadata": {}, "hash": "9e5fc09560edb7effbd9ab67a333c03eda5b98ed4acbc8cdf0de13133277290d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1254, "end_char_idx": 4229, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than
10X the amount of data.*

[Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by
Hugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five
different sizes: small, medium, large, xl and a distilled version of the small checkpoint: *distilgpt-2*.

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://openai.com/blog/better-language-models/).

## Usage tips

- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be
  observed in the *run_generation.py* example script.
- The model can take the *past_key_values* (for PyTorch) or *past* (for TF) as input, which is the previously computed
  key/value attention pairs. Using this (*past_key_values* or *past*) value prevents the model from re-computing
  pre-computed values in the context of text generation. For PyTorch, see *past_key_values* argument of the
  [`GPT2Model.forward`] method, or for TF the *past* argument of the
  [`TFGPT2Model.call`] method for more information on its usage.
- Enabling the *scale_attn_by_inverse_layer_idx* and *reorder_and_upcast_attn* flags will apply the training stability
  improvements from [Mistral](https://github.com/stanford-crfm/mistral/) (for PyTorch only).

## Usage example

The `generate()` method can be used to generate text using GPT2 model.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")

>>> prompt = "GPT2 is a model developed by OpenAI."

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).


key: _node_content
string_value:
{"id_": "5646deeb-9fae-4061-955e-8c8853e558b0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23c21ffd-fff2-4ac5-b292-68fd917442c7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "5d8f4f5ab7558d55de998f8a0c48985380ae7606db0f52835daea5d2e00d3ce4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dfa911f4-cea2-49fd-ad62-ba8dfe82134c", "node_type": "1", "metadata": {}, "hash": "813428da12c70079b3f413631e3a9d54dd1d9f554818584433187e584e95ae71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2592, "end_char_idx": 5876, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For PyTorch, see *past_key_values* argument of the
  [`GPT2Model.forward`] method, or for TF the *past* argument of the
  [`TFGPT2Model.call`] method for more information on its usage.
- Enabling the *scale_attn_by_inverse_layer_idx* and *reorder_and_upcast_attn* flags will apply the training stability
  improvements from [Mistral](https://github.com/stanford-crfm/mistral/) (for PyTorch only).

## Usage example

The `generate()` method can be used to generate text using GPT2 model.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")

>>> prompt = "GPT2 is a model developed by OpenAI."

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("gpt2", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
```


### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `gpt2` checkpoint and the Flash Attention 2 version of the model using a sequence length of 512.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/EduardoPacheco/documentation-images/resolve/main/gpt2_flash_attention_2_speedup.jpg">
</div>


key: _node_content
string_value:
{"id_": "dfa911f4-cea2-49fd-ad62-ba8dfe82134c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5646deeb-9fae-4061-955e-8c8853e558b0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "cc49e1c2393ac5bf979cce9ffc9f3e5c253769660a5fd577935b3e962bd495f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28cd0b23-7ca8-4b6b-a27f-f3f6e133415f", "node_type": "1", "metadata": {}, "hash": "3990f5f95ce583b47a4df6958c2ccdd47cac1ea2db295384b46798e9ea3109bb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4231, "end_char_idx": 7135, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("gpt2", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
```


### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `gpt2` checkpoint and the Flash Attention 2 version of the model using a sequence length of 512.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/EduardoPacheco/documentation-images/resolve/main/gpt2_flash_attention_2_speedup.jpg">
</div>


## Using Scaled Dot Product Attention (SDPA)
PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2", torch_dtype=torch.float16, attn_implementation="sdpa")
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with
[gpt2-large](https://huggingface.co/openai-community/gpt2-large), we saw the
following speedups during training and inference.


key: _node_content
string_value:
{"id_": "28cd0b23-7ca8-4b6b-a27f-f3f6e133415f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dfa911f4-cea2-49fd-ad62-ba8dfe82134c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "1f696c2265f8edebd6ec03d5daa62290059acc0f3538eacefe51b3e4e8e84196", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ebfb556f-27a8-49d5-b19d-47f4531ba0d6", "node_type": "1", "metadata": {}, "hash": "aa500fff1de9a4a5a23454332b8ea8924cfa71a69d2a4a1f8e659de5fbdd6c49", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5879, "end_char_idx": 8725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Using Scaled Dot Product Attention (SDPA)
PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2", torch_dtype=torch.float16, attn_implementation="sdpa")
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with
[gpt2-large](https://huggingface.co/openai-community/gpt2-large), we saw the
following speedups during training and inference.

### Training
| Batch size | Seq len |  Time per batch (Eager - s) | Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |
|-----------:|--------:|----------------------------:|--------------------------:|------------:|--------------------:|-------------------:|------------------:|
|          1 |     128 |                       0.039 |                     0.032 |      23.042 |             3482.32 |            3494.62 |            -0.352 |
|          1 |     256 |                       0.073 |                     0.059 |       25.15 |             3546.66 |             3552.6 |            -0.167 |
|          1 |     512 |                       0.155 |                     0.118 |       30.96 |              4230.1 |            3665.59 |              15.4 |
|          1 |    1024 |                       0.316 |                     0.209 |      50.839 |             8682.26 |            4881.09 |            77.875 |
|          2 |     128 |                        0.07 |                      0.06 |      15.324 |              3557.8 |            3545.91 |             0.335 |
|          2 |     256 |                       0.143 |                     0.122 |       16.53 |              3901.5 |            3657.68 |             6.666 |
|          2 |     512 |                       0.267 |                     0.213 |      25.626 |             7062.21 |            4876.47 |            44.822 |
|          2 |    1024 |                         OOM |                     0.404 |           / |                 OOM |            8096.


key: _node_content
string_value:
{"id_": "ebfb556f-27a8-49d5-b19d-47f4531ba0d6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28cd0b23-7ca8-4b6b-a27f-f3f6e133415f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "6633f199f3d590e898e23eb0248907889d797e4279053318f60cf03b21725492", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f831ca3-5fcf-41af-a639-0f939efa3942", "node_type": "1", "metadata": {}, "hash": "47019343afe82b5dc7d745c78f3acfc0a99e588645e3a0cb6c11a8bebaa7275c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7519, "end_char_idx": 10244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
039 |                     0.032 |      23.042 |             3482.32 |            3494.62 |            -0.352 |
|          1 |     256 |                       0.073 |                     0.059 |       25.15 |             3546.66 |             3552.6 |            -0.167 |
|          1 |     512 |                       0.155 |                     0.118 |       30.96 |              4230.1 |            3665.59 |              15.4 |
|          1 |    1024 |                       0.316 |                     0.209 |      50.839 |             8682.26 |            4881.09 |            77.875 |
|          2 |     128 |                        0.07 |                      0.06 |      15.324 |              3557.8 |            3545.91 |             0.335 |
|          2 |     256 |                       0.143 |                     0.122 |       16.53 |              3901.5 |            3657.68 |             6.666 |
|          2 |     512 |                       0.267 |                     0.213 |      25.626 |             7062.21 |            4876.47 |            44.822 |
|          2 |    1024 |                         OOM |                     0.404 |           / |                 OOM |            8096.35 | SDPA does not OOM |
|          4 |     128 |                       0.134 |                     0.128 |       4.412 |             3675.79 |            3648.72 |             0.742 |
|          4 |     256 |                       0.243 |                     0.217 |      12.292 |             6129.76 |            4871.12 |            25.839 |
|          4 |     512 |                       0.494 |                     0.406 |      21.687 |             12466.6 |            8102.64 |            53.858 |
|          4 |    1024 |                         OOM |                     0.795 |           / |                 OOM |            14568.2 | SDPA does not OOM |

### Inference
| Batch size | Seq len | Per token latency Eager (ms) | Per token latency SDPA (ms) | Speedup (%) | Mem Eager (MB) | Mem SDPA (MB) | Mem saved (%) |
|-----------:|--------:|-----------------------------:|----------------------------:|------------:|---------------:|--------------:|--------------:|
|          1 |     128 |                        7.991 |                       6.968 |      14.681 |         1685.2 |       1701.32 |        -0.947 |
|          1 |     256 |                        8.462 |                       7.199 |      17.536 |        1745.49 |       1770.78 |        -1.428 |
|          1 |     512 |                         8.68 |                       7.853 |      10.529 |        1907.69 |       1921.29 |        -0.708 |
|          1 |     768 |                        9.101 |                       8.365 |       8.


key: _node_content
string_value:
{"id_": "3f831ca3-5fcf-41af-a639-0f939efa3942", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ebfb556f-27a8-49d5-b19d-47f4531ba0d6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "05ecf144cd50956cde7813fec81a113c1b6c2677e1c9e7736b3e9989bea146f8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0679d3bf-519b-476c-a190-141864333563", "node_type": "1", "metadata": {}, "hash": "12acf598121ee8eadf85735bfbbfe4d38748e5fda0f93f832a75e7492c42ed04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8841, "end_char_idx": 11454, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
412 |             3675.79 |            3648.72 |             0.742 |
|          4 |     256 |                       0.243 |                     0.217 |      12.292 |             6129.76 |            4871.12 |            25.839 |
|          4 |     512 |                       0.494 |                     0.406 |      21.687 |             12466.6 |            8102.64 |            53.858 |
|          4 |    1024 |                         OOM |                     0.795 |           / |                 OOM |            14568.2 | SDPA does not OOM |

### Inference
| Batch size | Seq len | Per token latency Eager (ms) | Per token latency SDPA (ms) | Speedup (%) | Mem Eager (MB) | Mem SDPA (MB) | Mem saved (%) |
|-----------:|--------:|-----------------------------:|----------------------------:|------------:|---------------:|--------------:|--------------:|
|          1 |     128 |                        7.991 |                       6.968 |      14.681 |         1685.2 |       1701.32 |        -0.947 |
|          1 |     256 |                        8.462 |                       7.199 |      17.536 |        1745.49 |       1770.78 |        -1.428 |
|          1 |     512 |                         8.68 |                       7.853 |      10.529 |        1907.69 |       1921.29 |        -0.708 |
|          1 |     768 |                        9.101 |                       8.365 |       8.791 |        2032.93 |       2068.12 |        -1.701 |
|          2 |     128 |                        9.169 |                       9.001 |       1.861 |        1803.84 |        1811.4 |        -0.418 |
|          2 |     256 |                        9.907 |                        9.78 |       1.294 |        1907.72 |       1921.44 |        -0.714 |
|          2 |     512 |                       11.519 |                      11.644 |      -1.071 |        2176.86 |       2197.75 |        -0.951 |
|          2 |     768 |                       13.022 |                      13.407 |      -2.873 |         2464.3 |       2491.06 |        -1.074 |
|          4 |     128 |                       10.097 |                       9.831 |       2.709 |        1942.25 |       1985.13 |         -2.16 |
|          4 |     256 |                       11.599 |                      11.398 |       1.764 |        2177.28 |       2197.86 |        -0.937 |
|          4 |     512 |                       14.653 |                       14.45 |       1.411 |        2753.16 |       2772.57 |          -0.7 |
|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.


key: _node_content
string_value:
{"id_": "0679d3bf-519b-476c-a190-141864333563", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f831ca3-5fcf-41af-a639-0f939efa3942", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e8ad10cfc9b13e2e2dc6809d5d19edf1a60889269c5d6055c95c421aa8983f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ce65e59-b06a-441f-8363-a639c7a36906", "node_type": "1", "metadata": {}, "hash": "c1e143b8950968e7651beafcfef36155e5dc8a158284190cf6846d1c2bb73270", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10200, "end_char_idx": 12773, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
101 |                       8.365 |       8.791 |        2032.93 |       2068.12 |        -1.701 |
|          2 |     128 |                        9.169 |                       9.001 |       1.861 |        1803.84 |        1811.4 |        -0.418 |
|          2 |     256 |                        9.907 |                        9.78 |       1.294 |        1907.72 |       1921.44 |        -0.714 |
|          2 |     512 |                       11.519 |                      11.644 |      -1.071 |        2176.86 |       2197.75 |        -0.951 |
|          2 |     768 |                       13.022 |                      13.407 |      -2.873 |         2464.3 |       2491.06 |        -1.074 |
|          4 |     128 |                       10.097 |                       9.831 |       2.709 |        1942.25 |       1985.13 |         -2.16 |
|          4 |     256 |                       11.599 |                      11.398 |       1.764 |        2177.28 |       2197.86 |        -0.937 |
|          4 |     512 |                       14.653 |                       14.45 |       1.411 |        2753.16 |       2772.57 |          -0.7 |
|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.04 |       3343.97 |        -0.506 |




## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GPT2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-generation"/>

- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).
- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.
- A blog on [Training CodeParrot 🦜 from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.
- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.
- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.
- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb).


key: _node_content
string_value:
{"id_": "7ce65e59-b06a-441f-8363-a639c7a36906", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0679d3bf-519b-476c-a190-141864333563", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "c8fa34a85957facdd49f9769cf37a4a519b7f54a5dbb7b4651f3415d60e64a58", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f42f320-0a3d-463a-a23a-8e781300a661", "node_type": "1", "metadata": {}, "hash": "ec4de0dc4e01cdfe73437432d4bbe67267687c739604a5f35904b7f8ec1ee7fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11305, "end_char_idx": 14258, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
16 |       2772.57 |          -0.7 |
|          4 |     768 |                       17.846 |                      17.617 |       1.299 |        3327.04 |       3343.97 |        -0.506 |




## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GPT2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-generation"/>

- A blog on how to [Finetune a non-English GPT-2 Model with Hugging Face](https://www.philschmid.de/fine-tune-a-non-english-gpt-2-model-with-huggingface).
- A blog on [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate) with GPT-2.
- A blog on [Training CodeParrot 🦜 from Scratch](https://huggingface.co/blog/codeparrot), a large GPT-2 model.
- A blog on [Faster Text Generation with TensorFlow and XLA](https://huggingface.co/blog/tf-xla-generate) with GPT-2.
- A blog on [How to train a Language Model with Megatron-LM](https://huggingface.co/blog/megatron-training) with a GPT-2 model.
- A notebook on how to [finetune GPT2 to generate lyrics in the style of your favorite artist](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb). 🌎
- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). 🌎
- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the 🤗 Hugging Face Course.
- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).


key: _node_content
string_value:
{"id_": "7f42f320-0a3d-463a-a23a-8e781300a661", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89b4659-8bb4-4786-a637-d4af83954f55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ce65e59-b06a-441f-8363-a639c7a36906", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "31a8c05e1b50dbab2ffd6f7bf13fd5daf460ba68bab2286030b39bf750933653", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12774, "end_char_idx": 15876, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🌎
- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). 🌎
- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the 🤗 Hugging Face Course.
- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## GPT2Config

[[autodoc]] GPT2Config

## GPT2Tokenizer

[[autodoc]] GPT2Tokenizer
    - save_vocabulary

## GPT2TokenizerFast

[[autodoc]] GPT2TokenizerFast

## GPT2 specific outputs

[[autodoc]] models.gpt2.modeling_gpt2.GPT2DoubleHeadsModelOutput

[[autodoc]] models.gpt2.modeling_tf_gpt2.TFGPT2DoubleHeadsModelOutput

<frameworkcontent>
<pt>

## GPT2Model

[[autodoc]] GPT2Model
    - forward

## GPT2LMHeadModel

[[autodoc]] GPT2LMHeadModel
    - forward

## GPT2DoubleHeadsModel

[[autodoc]] GPT2DoubleHeadsModel
    - forward

## GPT2ForQuestionAnswering

[[autodoc]] GPT2ForQuestionAnswering
    - forward

## GPT2ForSequenceClassification

[[autodoc]] GPT2ForSequenceClassification
    - forward

## GPT2ForTokenClassification

[[autodoc]] GPT2ForTokenClassification
    - forward

</pt>
<tf>

## TFGPT2Model

[[autodoc]] TFGPT2Model
    - call

## TFGPT2LMHeadModel

[[autodoc]] TFGPT2LMHeadModel
    - call

## TFGPT2DoubleHeadsModel

[[autodoc]] TFGPT2DoubleHeadsModel
    - call

## TFGPT2ForSequenceClassification

[[autodoc]] TFGPT2ForSequenceClassification
    - call

## TFSequenceClassifierOutputWithPast

[[autodoc]] modeling_tf_outputs.TFSequenceClassifierOutputWithPast

## TFGPT2Tokenizer

[[autodoc]] TFGPT2Tokenizer

</tf>
<jax>

## FlaxGPT2Model

[[autodoc]] FlaxGPT2Model
    - __call__

## FlaxGPT2LMHeadModel

[[autodoc]] FlaxGPT2LMHeadModel
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e8563fb7-54f2-4664-98e4-bab96d7546c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a58ff85-1052-4568-9fb8-29d7b50d8dad", "node_type": "1", "metadata": {}, "hash": "97d15610b46e52c22c5aef59a729fd3425ebfbce03423e9ce4f3efa9ef32b83f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BROS

## Overview

The BROS model was proposed in [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://arxiv.org/abs/2108.04539) by Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, Sungrae Park.

BROS stands for *BERT Relying On Spatiality*. It is an encoder-only Transformer model that takes a sequence of tokens and their bounding boxes as inputs and outputs a sequence of hidden states. BROS encode relative spatial information instead of using absolute spatial information.

It is pre-trained with two objectives: a token-masked language modeling objective (TMLM) used in BERT, and a novel area-masked language modeling objective (AMLM)
In TMLM, tokens are randomly masked, and the model predicts the masked tokens using spatial information and other unmasked tokens.
AMLM is a 2D version of TMLM. It randomly masks text tokens and predicts with the same information as TMLM, but it masks text blocks (areas).

`BrosForTokenClassification` has a simple linear layer on top of BrosModel. It predicts the label of each token.
`BrosSpadeEEForTokenClassification` has an `initial_token_classifier` and `subsequent_token_classifier` on top of BrosModel. `initial_token_classifier` is used to predict the first token of each entity, and `subsequent_token_classifier` is used to predict the next token of within entity. `BrosSpadeELForTokenClassification` has an `entity_linker` on top of BrosModel. `entity_linker` is used to predict the relation between two entities.

`BrosForTokenClassification` and `BrosSpadeEEForTokenClassification` essentially perform the same job. However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.

`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.

BROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.

The abstract from the paper is the following:

*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.


key: _node_content
string_value:
{"id_": "4a58ff85-1052-4568-9fb8-29d7b50d8dad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8563fb7-54f2-4664-98e4-bab96d7546c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "0b5d6965c6205045e767e448d585a8eb2337ebc11d893b94e68ced61dfcca0ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e01b30c7-b18d-455c-9e05-2d51bdda3b29", "node_type": "1", "metadata": {}, "hash": "a51d28b889aa12b15039a238fa0681ba8d697c6c02ef2efef96e03139551ee02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1663, "end_char_idx": 4829, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.

`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.

BROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.

The abstract from the paper is the following:

*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*

This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).

## Usage tips and examples

- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.

```python
def expand_and_normalize_bbox(bboxes, doc_width, doc_height):
    # here, bboxes are numpy array

    # Normalize bbox -> 0 ~ 1
    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width
    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height
```

- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,


key: _node_content
string_value:
{"id_": "e01b30c7-b18d-455c-9e05-2d51bdda3b29", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef81e749-3739-47bf-8763-887fc68ff8ed", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a58ff85-1052-4568-9fb8-29d7b50d8dad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "0a666a59f58a765d86aa516e27753bd154e4422ea8b740ffe0422bd69eea40be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3298, "end_char_idx": 6382, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*

This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).

## Usage tips and examples

- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.

```python
def expand_and_normalize_bbox(bboxes, doc_width, doc_height):
    # here, bboxes are numpy array

    # Normalize bbox -> 0 ~ 1
    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width
    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height
```

- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,


```python
def make_box_first_token_mask(bboxes, words, tokenizer, max_seq_length=512):

    box_first_token_mask = np.zeros(max_seq_length, dtype=np.bool_)

    # encode(tokenize) each word from words (List[str])
    input_ids_list: List[List[int]] = [tokenizer.encode(e, add_special_tokens=False) for e in words]

    # get the length of each box
    tokens_length_list: List[int] = [len(l) for l in input_ids_list]

    box_end_token_indices = np.array(list(itertools.accumulate(tokens_length_list)))
    box_start_token_indices = box_end_token_indices - np.array(tokens_length_list)

    # filter out the indices that are out of max_seq_length
    box_end_token_indices = box_end_token_indices[box_end_token_indices < max_seq_length - 1]
    if len(box_start_token_indices) > len(box_end_token_indices):
        box_start_token_indices = box_start_token_indices[: len(box_end_token_indices)]

    # set box_start_token_indices to True
    box_first_token_mask[box_start_token_indices] = True

    return box_first_token_mask

```

## Resources

- Demo scripts can be found [here](https://github.com/clovaai/bros).

## BrosConfig

[[autodoc]] BrosConfig

## BrosProcessor

[[autodoc]] BrosProcessor
    - __call__

## BrosModel

[[autodoc]] BrosModel
    - forward


## BrosForTokenClassification

[[autodoc]] BrosForTokenClassification
    - forward

## BrosSpadeEEForTokenClassification

[[autodoc]] BrosSpadeEEForTokenClassification
    - forward

## BrosSpadeELForTokenClassification

[[autodoc]] BrosSpadeELForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "f54533f1-d071-4fa3-b34c-af2c405606cc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e571e305-2449-41c7-863c-d8325947cee2", "node_type": "1", "metadata": {}, "hash": "7e9a9bea7f2e43028c15abf364e8ff9680e7e382d6c64fa6eb7e1b10a403b0dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3006, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoBERTa

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=roberta">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-roberta-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/roberta-base">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
<a href="https://huggingface.co/papers/1907.11692">
<img alt="Paper page" src="https://img.shields.io/badge/Paper%20page-1907.11692-green">
</a>
</div>

## Overview

The RoBERTa model was proposed in [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, [Myle Ott](https://huggingface.co/myleott), Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018.

It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with
much larger mini-batches and learning rates.

The abstract from the paper is the following:

*Language model pretraining has led to significant performance gains but careful comparison between different
approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes,
and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication
study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and
training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every
model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results
highlight the importance of previously overlooked design choices, and raise questions about the source of recently
reported improvements. We release our models and code.*

This model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).

## Usage tips

- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup
  for RoBERTa pretrained models.
- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
  different pretraining scheme.
- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just
  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).
- RoBERTa is similar to BERT but with better pretraining techniques:

    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.
    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).
    * Larger batches: Training uses larger batches.


key: _node_content
string_value:
{"id_": "e571e305-2449-41c7-863c-d8325947cee2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f54533f1-d071-4fa3-b34c-af2c405606cc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "f919e9d84035c7557106fe58a50fae2bb5596a128f5914b320741e3d14102c87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3b7e37f-0904-4b67-b943-fae9afb3b3ca", "node_type": "1", "metadata": {}, "hash": "795a632c853f71615069acd4102ba841a2189e49b056d2a7ce1b461bb4899719", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1411, "end_char_idx": 4760, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We present a replication
study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and
training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every
model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results
highlight the importance of previously overlooked design choices, and raise questions about the source of recently
reported improvements. We release our models and code.*

This model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).

## Usage tips

- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup
  for RoBERTa pretrained models.
- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
  different pretraining scheme.
- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just
  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).
- RoBERTa is similar to BERT but with better pretraining techniques:

    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.
    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).
    * Larger batches: Training uses larger batches.
    * Byte-level BPE vocabulary: Uses BPE with bytes as a subunit instead of characters, accommodating Unicode characters.
- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).
- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.
- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). 🌎
- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).


key: _node_content
string_value:
{"id_": "a3b7e37f-0904-4b67-b943-fae9afb3b3ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e571e305-2449-41c7-863c-d8325947cee2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5fcf0966c0c2522aa49518d5dec924edefff4264ba581f7f50e961b2fe5c6faa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2525c8b-df68-49de-bcce-24c3096c4677", "node_type": "1", "metadata": {}, "hash": "dff54f4b19bb58959515f781b6e03fca32540dc10c7c42313b3d29bfd90cf894", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3130, "end_char_idx": 6308, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).
- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.
- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). 🌎
- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa.


key: _node_content
string_value:
{"id_": "c2525c8b-df68-49de-bcce-24c3096c4677", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a3b7e37f-0904-4b67-b943-fae9afb3b3ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "1c8edd7cd0038c9ff98768bf24ce02978a963097e08e9033a1973f66eea6b5ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "725825e8-fd6a-420c-9a4c-ed326c52422a", "node_type": "1", "metadata": {}, "hash": "778595b00b96a9a20dc44777b666f8f07b8ce664384bef41d5348caabfbd1321", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4761, "end_char_idx": 7927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa.
- [`RobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.
- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).


key: _node_content
string_value:
{"id_": "725825e8-fd6a-420c-9a4c-ed326c52422a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2525c8b-df68-49de-bcce-24c3096c4677", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "e4ee4ab04e6d3ac9e114bfae173a82207e06a76022c96955f213bfba316f3d50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2feb17f2-7bfa-40d8-81b9-0fcfc7ff77cb", "node_type": "1", "metadata": {}, "hash": "d3f3a2e5f89a5f00130aaf662421b136792d37673c33ead1919633573853632d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6636, "end_char_idx": 9144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.
- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**
- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).


key: _node_content
string_value:
{"id_": "2feb17f2-7bfa-40d8-81b9-0fcfc7ff77cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c6eb115-c5bd-4197-94a5-49da19a503aa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "725825e8-fd6a-420c-9a4c-ed326c52422a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "d170bbc4843d16505b2b2d462c8b143ddb0e41eff1604afe9b61b88f199bf288", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7641, "end_char_idx": 11330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**
- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

## RobertaConfig

[[autodoc]] RobertaConfig

## RobertaTokenizer

[[autodoc]] RobertaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## RobertaTokenizerFast

[[autodoc]] RobertaTokenizerFast
    - build_inputs_with_special_tokens

<frameworkcontent>
<pt>

## RobertaModel

[[autodoc]] RobertaModel
    - forward

## RobertaForCausalLM

[[autodoc]] RobertaForCausalLM
    - forward

## RobertaForMaskedLM

[[autodoc]] RobertaForMaskedLM
    - forward

## RobertaForSequenceClassification

[[autodoc]] RobertaForSequenceClassification
    - forward

## RobertaForMultipleChoice

[[autodoc]] RobertaForMultipleChoice
    - forward

## RobertaForTokenClassification

[[autodoc]] RobertaForTokenClassification
    - forward

## RobertaForQuestionAnswering

[[autodoc]] RobertaForQuestionAnswering
    - forward

</pt>
<tf>

## TFRobertaModel

[[autodoc]] TFRobertaModel
    - call

## TFRobertaForCausalLM

[[autodoc]] TFRobertaForCausalLM
    - call

## TFRobertaForMaskedLM

[[autodoc]] TFRobertaForMaskedLM
    - call

## TFRobertaForSequenceClassification

[[autodoc]] TFRobertaForSequenceClassification
    - call

## TFRobertaForMultipleChoice

[[autodoc]] TFRobertaForMultipleChoice
    - call

## TFRobertaForTokenClassification

[[autodoc]] TFRobertaForTokenClassification
    - call

## TFRobertaForQuestionAnswering

[[autodoc]] TFRobertaForQuestionAnswering
    - call

</tf>
<jax>

## FlaxRobertaModel

[[autodoc]] FlaxRobertaModel
    - __call__

## FlaxRobertaForCausalLM

[[autodoc]] FlaxRobertaForCausalLM
    - __call__

## FlaxRobertaForMaskedLM

[[autodoc]] FlaxRobertaForMaskedLM
    - __call__

## FlaxRobertaForSequenceClassification

[[autodoc]] FlaxRobertaForSequenceClassification
    - __call__

## FlaxRobertaForMultipleChoice

[[autodoc]] FlaxRobertaForMultipleChoice
    - __call__

## FlaxRobertaForTokenClassification

[[autodoc]] FlaxRobertaForTokenClassification
    - __call__

## FlaxRobertaForQuestionAnswering

[[autodoc]] FlaxRobertaForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "5a8142fc-ee19-4a71-ba56-4084fa5ff6dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/swinv2", "title": "Swin Transformer V2", "tokens": 894, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4e36004e-21c0-44be-8f8b-4d5d0f2ffd53", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/swinv2", "title": "Swin Transformer V2", "tokens": 894, "retrieve_doc": true, "source": "transformers"}, "hash": "6cdd98d6b1a403b4f727ab32240655dd8b9094f2947e7a6d4566035fd54ffb19", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3253, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Swin Transformer V2

## Overview

The Swin Transformer V2 model was proposed in [Swin Transformer V2: Scaling Up Capacity and Resolution](https://arxiv.org/abs/2111.09883) by Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo.

The abstract from the paper is the following:

*Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536×1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time.*

This model was contributed by [nandwalritik](https://huggingface.co/nandwalritik).
The original code can be found [here](https://github.com/microsoft/Swin-Transformer).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Swin Transformer v2.

<PipelineTag pipeline="image-classification"/>

- [`Swinv2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

Besides that:

- [`Swinv2ForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## Swinv2Config

[[autodoc]] Swinv2Config

## Swinv2Model

[[autodoc]] Swinv2Model
    - forward

## Swinv2ForMaskedImageModeling

[[autodoc]] Swinv2ForMaskedImageModeling
    - forward

## Swinv2ForImageClassification

[[autodoc]] transformers.Swinv2ForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "358f7768-eca1-4ad3-b9d6-2fe9c921b8bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9573053-9509-4583-b226-9f957fc37593", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "hash": "6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d02f2d15-418e-45cc-b6b6-47e11b26dbd6", "node_type": "1", "metadata": {}, "hash": "4f2a5129568f2763e46b6b15dad6e397bc22d863f8988e1eacbfaf6ce921818b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LED

## Overview

The LED model was proposed in [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz
Beltagy, Matthew E. Peters, Arman Cohan.

The abstract from the paper is the following:

*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales
quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention
mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or
longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local
windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we
evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In
contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on
WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting
long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization
dataset.*

## Usage tips

- [`LEDForConditionalGeneration`] is an extension of
  [`BartForConditionalGeneration`] exchanging the traditional *self-attention* layer with
  *Longformer*'s *chunked self-attention* layer. [`LEDTokenizer`] is an alias of
  [`BartTokenizer`].
- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of
  1024 tokens.
- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is
  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.
- LED makes use of *global attention* by means of the `global_attention_mask` (see
  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first
  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.
- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)
  errors. This can be done by executing `model.gradient_checkpointing_enable()`. 
 Moreover, the `use_cache=False`
  flag can be used to disable the caching mechanism to save memory.
- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

## Resources

- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).
- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).


key: _node_content
string_value:
{"id_": "d02f2d15-418e-45cc-b6b6-47e11b26dbd6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9573053-9509-4583-b226-9f957fc37593", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "hash": "6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "358f7768-eca1-4ad3-b9d6-2fe9c921b8bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f4db522a042a98ba9f0700d1e9f7c98553a052f1a26f5be81c0ec6fa890f22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1597, "end_char_idx": 4619, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`LEDTokenizer`] is an alias of
  [`BartTokenizer`].
- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of
  1024 tokens.
- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is
  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.
- LED makes use of *global attention* by means of the `global_attention_mask` (see
  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first
  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.
- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)
  errors. This can be done by executing `model.gradient_checkpointing_enable()`. 
 Moreover, the `use_cache=False`
  flag can be used to disable the caching mechanism to save memory.
- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

## Resources

- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).
- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).
- [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## LEDConfig

[[autodoc]] LEDConfig

## LEDTokenizer

[[autodoc]] LEDTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## LEDTokenizerFast

[[autodoc]] LEDTokenizerFast

## LED specific outputs

[[autodoc]] models.led.modeling_led.LEDEncoderBaseModelOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqModelOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqLMOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqSequenceClassifierOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqQuestionAnsweringModelOutput

[[autodoc]] models.led.modeling_tf_led.TFLEDEncoderBaseModelOutput

[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqModelOutput

[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqLMOutput

<frameworkcontent>
<pt>

## LEDModel

[[autodoc]] LEDModel
    - forward

## LEDForConditionalGeneration

[[autodoc]] LEDForConditionalGeneration
    - forward

## LEDForSequenceClassification

[[autodoc]] LEDForSequenceClassification
    - forward

## LEDForQuestionAnswering

[[autodoc]] LEDForQuestionAnswering
    - forward

</pt>
<tf>

## TFLEDModel

[[autodoc]] TFLEDModel
    - call

## TFLEDForConditionalGeneration

[[autodoc]] TFLEDForConditionalGeneration
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "ea4ab563-b980-4fa4-bbdf-35809b1c3130", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "hash": "bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2d588e9-690b-4294-8917-998650565143", "node_type": "1", "metadata": {}, "hash": "3f0fb0bf75e05b5df4469bf46b388c65e2e7dd434605d83756abf6a6b9c48d61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3023, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Phi-3

## Overview

The Phi-3 model was proposed in [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://arxiv.org/abs/2404.14219) by Microsoft.

### Summary

The abstract from the Phi-3 paper is the following:

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).

The original code for Phi-3 can be found [here](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct).

## Usage tips

- This model is very similar to `Llama` with the main difference of [`Phi3SuScaledRotaryEmbedding`] and [`Phi3YarnScaledRotaryEmbedding`], where they are used to extend the context of the rotary embeddings. The query, key and values are fused, and the MLP's up and gate projection layers are also fused.
- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.

## How to use Phi-3

<Tip warning={true}>

Phi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:

* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.

* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.

</Tip>

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

>>> messages = [{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}]
>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

>>> outputs = model.generate(inputs, max_new_tokens=32)
>>> text = tokenizer.batch_decode(outputs)[0]
>>> print(text)
<s><|user|> 
Can you provide ways to eat combinations of bananas and dragonfruits?<|end|> 
<|assistant|> 
Certainly! Bananas and dragonfruits can be combined in various delicious ways.


key: _node_content
string_value:
{"id_": "e2d588e9-690b-4294-8917-998650565143", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "45471918-bdde-4ca9-9e7b-5b003a8f8525", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "hash": "bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea4ab563-b980-4fa4-bbdf-35809b1c3130", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "hash": "23f8da943fcf1bf639bae95c6dd74a7f324beac8f61e3b19049bb70c83c8913f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1434, "end_char_idx": 3482, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The query, key and values are fused, and the MLP's up and gate projection layers are also fused.
- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.

## How to use Phi-3

<Tip warning={true}>

Phi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:

* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.

* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.

</Tip>

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

>>> messages = [{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}]
>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

>>> outputs = model.generate(inputs, max_new_tokens=32)
>>> text = tokenizer.batch_decode(outputs)[0]
>>> print(text)
<s><|user|> 
Can you provide ways to eat combinations of bananas and dragonfruits?<|end|> 
<|assistant|> 
Certainly! Bananas and dragonfruits can be combined in various delicious ways. Here are some ideas for eating combinations of bananas and
```

## Phi3Config

[[autodoc]] Phi3Config

<frameworkcontent>
<pt>

## Phi3Model

[[autodoc]] Phi3Model
    - forward

## Phi3ForCausalLM

[[autodoc]] Phi3ForCausalLM
    - forward
    - generate

## Phi3ForSequenceClassification

[[autodoc]] Phi3ForSequenceClassification
    - forward

## Phi3ForTokenClassification

[[autodoc]] Phi3ForTokenClassification
    - forward

</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "6a0006f6-1752-4aef-826b-2ef38e61320f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "246a192f-4dec-4e70-a40a-a12ff7dace69", "node_type": "1", "metadata": {}, "hash": "3468b4d14c4f66f7d2e223f78298ebb162aeca0c46267819de0f7399ae3dcf4d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3143, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# UPerNet

## Overview

The UPerNet model was proposed in [Unified Perceptual Parsing for Scene Understanding](https://arxiv.org/abs/1807.10221)
by Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun. UPerNet is a general framework to effectively segment
a wide range of concepts from images, leveraging any vision backbone like [ConvNeXt](convnext) or [Swin](swin).

The abstract from the paper is the following:

*Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/upernet_architecture.jpg"
alt="drawing" width="600"/>

<small> UPerNet framework. Taken from the <a href="https://arxiv.org/abs/1807.10221">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).

## Usage examples

UPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:

```py
from transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = SwinConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

To use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:

```py
from transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = ConvNextConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

Note that this will randomly initialize all the weights of the model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with UPerNet.

- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).
- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb).


key: _node_content
string_value:
{"id_": "246a192f-4dec-4e70-a40a-a12ff7dace69", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "721f362b-24da-47bb-90a5-c118c9addf2a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a0006f6-1752-4aef-826b-2ef38e61320f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "38138b6f220a5f7ad2bf742f55a19813bb1c080fdc9850e3b7342f260765aa71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1542, "end_char_idx": 3588, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).

## Usage examples

UPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:

```py
from transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = SwinConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

To use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:

```py
from transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = ConvNextConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

Note that this will randomly initialize all the weights of the model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with UPerNet.

- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).
- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb).
- See also: [Semantic segmentation task guide](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## UperNetConfig

[[autodoc]] UperNetConfig

## UperNetForSemanticSegmentation

[[autodoc]] UperNetForSemanticSegmentation
    - forward


key: _node_content
string_value:
{"id_": "6f080e47-8b11-4a33-ad52-8f0fde3ada87", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17f37e06-75ea-443e-994d-1126855f60b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "hash": "36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e475039c-88e3-4d47-b206-d63db89d2963", "node_type": "1", "metadata": {}, "hash": "1671e10b84b6b2ba565fc0072b3d01791f6449488c59a86f81a1d80eef670c47", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Blenderbot

## Overview

The Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,
Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.

The abstract of the paper is the following:

*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that
scaling neural models in the number of parameters and the size of the data they are trained on gives improved results,
we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of
skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to
their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent
persona. We show that large scale models can learn these skills when given appropriate training data and choice of
generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models
and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn
dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing
failure cases of our models.*

This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/facebookresearch/ParlAI) .

## Usage tips and example

Blenderbot is a model with absolute position embeddings so it's usually advised to pad the inputs on the right 
rather than the left.

An example:

```python
>>> from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration

>>> mname = "facebook/blenderbot-400M-distill"
>>> model = BlenderbotForConditionalGeneration.from_pretrained(mname)
>>> tokenizer = BlenderbotTokenizer.from_pretrained(mname)
>>> UTTERANCE = "My friends are cool but they eat too many carbs."
>>> inputs = tokenizer([UTTERANCE], return_tensors="pt")
>>> reply_ids = model.generate(**inputs)
>>> print(tokenizer.batch_decode(reply_ids))
["<s> That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?</s>"]
```

## Implementation Notes

- Blenderbot uses a standard [seq2seq model transformer](https://arxiv.org/pdf/1706.03762.pdf) based architecture.
- Available checkpoints can be found in the [model hub](https://huggingface.co/models?search=blenderbot).
- This is the *default* Blenderbot model class. However, some smaller checkpoints, such as
  `facebook/blenderbot_small_90M`, have a different architecture and consequently should be used with
  [BlenderbotSmall](blenderbot-small).

  
## Resources

- [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## BlenderbotConfig

[[autodoc]] BlenderbotConfig

## BlenderbotTokenizer

[[autodoc]] BlenderbotTokenizer
    - build_inputs_with_special_tokens

## BlenderbotTokenizerFast

[[autodoc]] BlenderbotTokenizerFast
    - build_inputs_with_special_tokens


key: _node_content
string_value:
{"id_": "e475039c-88e3-4d47-b206-d63db89d2963", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17f37e06-75ea-443e-994d-1126855f60b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "hash": "36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f080e47-8b11-4a33-ad52-8f0fde3ada87", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "hash": "086afa922d1464c5c9f1c1a4180fe3020cf6b8ead67cd6a88a0969880d5d6cff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3297, "end_char_idx": 4181, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>

## BlenderbotModel

See [`~transformers.BartModel`] for arguments to *forward* and *generate*

[[autodoc]] BlenderbotModel
    - forward

## BlenderbotForConditionalGeneration

See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* and *generate*

[[autodoc]] BlenderbotForConditionalGeneration
    - forward

## BlenderbotForCausalLM

[[autodoc]] BlenderbotForCausalLM
    - forward

</pt>
<tf>

## TFBlenderbotModel

[[autodoc]] TFBlenderbotModel
    - call

## TFBlenderbotForConditionalGeneration

[[autodoc]] TFBlenderbotForConditionalGeneration
    - call

</tf>
<jax>

## FlaxBlenderbotModel

[[autodoc]] FlaxBlenderbotModel
    - __call__
    - encode
    - decode

## FlaxBlenderbotForConditionalGeneration

[[autodoc]] FlaxBlenderbotForConditionalGeneration
    - __call__
    - encode
    - decode

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "08abde3f-0b2f-4636-b5e6-85bd6fb8fb2b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "hash": "c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f80487b-75a6-45fc-af8c-217bccfa48c8", "node_type": "1", "metadata": {}, "hash": "7c86b7e34c683615d4aa424eaea5958ee97a6b3d0998581f868d7d53f1585a1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2871, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Splinter

## Overview

The Splinter model was proposed in [Few-Shot Question Answering by Pretraining Span Selection](https://arxiv.org/abs/2101.00438) by Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy. Splinter
is an encoder-only transformer (similar to BERT) pretrained using the recurring span selection task on a large corpus
comprising Wikipedia and the Toronto Book Corpus.

The abstract from the paper is the following:

In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order
of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred
training examples are available, and observe that standard models perform poorly, highlighting the discrepancy between
current pretraining objectives and question answering. We propose a new pretraining scheme tailored for question
answering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all
recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans
are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select
the answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD
with only 128 training examples), while maintaining competitive performance in the high-resource setting.

This model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).

## Usage tips

- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize
  to question representations which are used to predict the answers. This layer is called QASS, and is the default
  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:
- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already
  contains this special token. Also, its default behavior is to use this token when two sequences are given (for
  example, in the *run_qa.py* script).
- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for
  the success of your model, especially in a few-shot setting.
- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that
  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one
  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at
  fine-tuning, as it is shown to yield better results for some cases in the paper.


key: _node_content
string_value:
{"id_": "4f80487b-75a6-45fc-af8c-217bccfa48c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "655aebe8-899a-4122-bcf6-e14bf96a21d0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "hash": "c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "08abde3f-0b2f-4636-b5e6-85bd6fb8fb2b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "hash": "a51b64a63b5822aa0dd685583af29628804befb10d84db6ab5442ab5a36d7068", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1291, "end_char_idx": 3469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD
with only 128 training examples), while maintaining competitive performance in the high-resource setting.

This model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).

## Usage tips

- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize
  to question representations which are used to predict the answers. This layer is called QASS, and is the default
  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:
- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already
  contains this special token. Also, its default behavior is to use this token when two sequences are given (for
  example, in the *run_qa.py* script).
- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for
  the success of your model, especially in a few-shot setting.
- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that
  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one
  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at
  fine-tuning, as it is shown to yield better results for some cases in the paper.

## Resources

- [Question answering task guide](../tasks/question-answering)

## SplinterConfig

[[autodoc]] SplinterConfig

## SplinterTokenizer

[[autodoc]] SplinterTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## SplinterTokenizerFast

[[autodoc]] SplinterTokenizerFast

## SplinterModel

[[autodoc]] SplinterModel
    - forward

## SplinterForQuestionAnswering

[[autodoc]] SplinterForQuestionAnswering
    - forward

## SplinterForPreTraining

[[autodoc]] SplinterForPreTraining
    - forward


key: _node_content
string_value:
{"id_": "1901e493-74ed-45e6-b842-8236bd253ac3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sew-d", "title": "SEW-D", "tokens": 637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0f3d6593-9057-4868-958f-c4cfc451f1bd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sew-d", "title": "SEW-D", "tokens": 637, "retrieve_doc": true, "source": "transformers"}, "hash": "7932e84258bad487dcc35ad86f38719dc0750fa3f4768f89de9adc2e5acbcf9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SEW-D

## Overview

SEW-D (Squeezed and Efficient Wav2Vec with Disentangled attention) was proposed in [Performance-Efficiency Trade-offs
in Unsupervised Pre-training for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim,
Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.

The abstract from the paper is the following:

*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition
(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance
and its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a
pre-trained model architecture with significant improvements along both performance and efficiency dimensions across a
variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x
inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference
time, SEW reduces word error rate by 25-50% across different model sizes.*

This model was contributed by [anton-l](https://huggingface.co/anton-l).

## Usage tips

- SEW-D is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- SEWDForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].

## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## SEWDConfig

[[autodoc]] SEWDConfig

## SEWDModel

[[autodoc]] SEWDModel
    - forward

## SEWDForCTC

[[autodoc]] SEWDForCTC
    - forward

## SEWDForSequenceClassification

[[autodoc]] SEWDForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "1eaae299-918c-4640-afce-9cccb64b285c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0231f515-d82e-41cf-a8cc-958d9c3387d4", "node_type": "1", "metadata": {}, "hash": "b8e74e9d13d72f4bf23603a5fc8ef2560b7c5797615084d281d7ce1ee71f0eb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TAPAS

## Overview

The TAPAS model was proposed in [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://www.aclweb.org/anthology/2020.acl-main.398)
by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and Julian Martin Eisenschlos. It's a BERT-based model specifically 
designed (and pre-trained) for answering questions about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 
token types that encode tabular structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising 
millions of tables from English Wikipedia and corresponding texts. 

For question answering, TAPAS has 2 heads on top: a cell selection head and an aggregation head, for (optionally) performing aggregations (such as counting or summing) among selected cells. TAPAS has been fine-tuned on several datasets: 
- [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)
- [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)
- [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce). 

It achieves state-of-the-art on both SQA and WTQ, while having comparable performance to SOTA on WikiSQL, with a much simpler architecture.

The abstract from the paper is the following:

*Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*

In addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task).


key: _node_content
string_value:
{"id_": "0231f515-d82e-41cf-a8cc-958d9c3387d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1eaae299-918c-4640-afce-9cccb64b285c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "d6b3cdf8449415b8bbe646328e23bedbdef5b9b067de10e76e87a9d3965719f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b5d30ca-f7e4-478d-9f6a-079a866be146", "node_type": "1", "metadata": {}, "hash": "54f75347f167662c9e9b12b575e19602357874423cd0452b5e98e5f7693efd94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1628, "end_char_idx": 5153, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*

In addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task). For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png"
alt="drawing" width="600"/> 

<small> TAPAS architecture. Taken from the <a href="https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html">original blog post</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).

## Usage tips

- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision="no_reset"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.


key: _node_content
string_value:
{"id_": "0b5d30ca-f7e4-478d-9f6a-079a866be146", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0231f515-d82e-41cf-a8cc-958d9c3387d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "c5c67b023e5d77d4ef4825f5cda00f03a40a730ffba7398b1d44095406639359", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8137dc2f-8b41-4c8e-908f-fdd9710df52f", "node_type": "1", "metadata": {}, "hash": "9e7e7fff44098dce7ab89839730c10a6d7350eb0dbf2808b4123757f6337765e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3458, "end_char_idx": 6633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png"
alt="drawing" width="600"/> 

<small> TAPAS architecture. Taken from the <a href="https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html">original blog post</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).

## Usage tips

- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision="no_reset"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.
- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).
- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as "what is his age?" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See "Usage" section for more info.
- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.

## Usage: fine-tuning

Here we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.


key: _node_content
string_value:
{"id_": "8137dc2f-8b41-4c8e-908f-fdd9710df52f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b5d30ca-f7e4-478d-9f6a-079a866be146", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "53b4fd07ba325d6af22ef1bdf229cf88a811f6466c00b0f19f5e20000ad0dc14", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "919ff337-caff-41aa-a204-9796c9436568", "node_type": "1", "metadata": {}, "hash": "5853700e59e5aaa95a01af1e6d806a23720f14a9e488a4c321261854b4657eb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5070, "end_char_idx": 8015, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that it's usually advised to pad the inputs on the right rather than the left.
- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).
- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as "what is his age?" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See "Usage" section for more info.
- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.

## Usage: fine-tuning

Here we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.

**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**

Basically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:

1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask "what's the name of the first actor?" then you can ask a follow-up question such as "how old is he?". Here, questions do not involve any aggregation (all questions are cell selection questions).
2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask "what's the total number of goals Cristiano Ronaldo made in his career?". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.
3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.


key: _node_content
string_value:
{"id_": "919ff337-caff-41aa-a204-9796c9436568", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8137dc2f-8b41-4c8e-908f-fdd9710df52f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "815cca00555d4e49fb99d6f06cd5627dfd7d9a9b09a739e7cefab84e6b0e10d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9097cdfa-94dd-4fb3-baae-5b117d51031d", "node_type": "1", "metadata": {}, "hash": "07c2ee055ce6f362a7410769e0cd6fd2cd1852b46c1f2867f2f2e1141acb5fcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6286, "end_char_idx": 10123, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.

## Usage: fine-tuning

Here we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.

**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**

Basically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:

1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask "what's the name of the first actor?" then you can ask a follow-up question such as "how old is he?". Here, questions do not involve any aggregation (all questions are cell selection questions).
2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask "what's the total number of goals Cristiano Ronaldo made in his career?". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.
3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.

To summarize:

| **Task**                            | **Example dataset** | **Description**                                                                                         |
|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|
| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |
| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |

<frameworkcontent>
<pt>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way.


key: _node_content
string_value:
{"id_": "9097cdfa-94dd-4fb3-baae-5b117d51031d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "919ff337-caff-41aa-a204-9796c9436568", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "f4f4f0d59938eaf215ec69c506e83cc666ba1a7ab549265cf9d18100f1754588", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6e79ec25-5c74-41bb-969b-1b9390be12fa", "node_type": "1", "metadata": {}, "hash": "6bae031756ce043224ba32faee981850fd67269f509027fa69a07b2471954c4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8017, "end_char_idx": 11784, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To summarize:

| **Task**                            | **Example dataset** | **Description**                                                                                         |
|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|
| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |
| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |

<frameworkcontent>
<pt>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</pt>
<tf>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration.


key: _node_content
string_value:
{"id_": "6e79ec25-5c74-41bb-969b-1b9390be12fa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9097cdfa-94dd-4fb3-baae-5b117d51031d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "a2f98c5841119cccaec3faf68f8769ed6e19592332ca63edf45023e853fcdcd0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "197de4b7-4c2f-4fb6-9bec-a94ac738425e", "node_type": "1", "metadata": {}, "hash": "729ddea90a9e793940fcc8f071cd5c8cec97e6b6c3d3641c1020104c276daeeb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10124, "end_char_idx": 13467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here's an example:

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</pt>
<tf>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</tf>
</frameworkcontent>

What you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.

For a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).

**STEP 2: Prepare your data in the SQA format**

Second, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:

- `id`: optional, id of the table-question pair, for bookkeeping purposes.
- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA).


key: _node_content
string_value:
{"id_": "197de4b7-4c2f-4fb6-9bec-a94ac738425e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e79ec25-5c74-41bb-969b-1b9390be12fa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "27ff86a8e9e17bf77ea19de0b8d828369a12cf2267cb19533f4965482377d8c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35b5cb87-a786-4c60-82af-335a6bdf07a8", "node_type": "1", "metadata": {}, "hash": "05338a496a08ff547ae8a53b5860be98e6963b991e8c251e144094cd293cd2fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11785, "end_char_idx": 14917, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</tf>
</frameworkcontent>

What you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.

For a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).

**STEP 2: Prepare your data in the SQA format**

Second, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:

- `id`: optional, id of the table-question pair, for bookkeeping purposes.
- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.
- `question`: string
- `table_file`: string, name of a csv file containing the tabular data
- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)
- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)
- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)
- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)

The tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.


key: _node_content
string_value:
{"id_": "35b5cb87-a786-4c60-82af-335a6bdf07a8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "197de4b7-4c2f-4fb6-9bec-a94ac738425e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "ef8ce7b6fc8cd8e36015416868943c87a105d5d92367cff769a952cc9e61d992", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3feb354d-a3fa-48d8-91b5-f76e117d1fe1", "node_type": "1", "metadata": {}, "hash": "2ad94ff4bcb398e41591ae5d15388101156198403251862b34b3b4bf1f7e1630", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13205, "end_char_idx": 16801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.
- `question`: string
- `table_file`: string, name of a csv file containing the tabular data
- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)
- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)
- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)
- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)

The tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.

**STEP 3: Convert your data into tensors using TapasTokenizer**

<frameworkcontent>
<pt>
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different
inputs to be fine-tuned:

| **Task**                           | **Required inputs**                                                                                                 |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |
|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |
| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |

[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:

```py
>>> from transformers import TapasTokenizer
>>> import pandas as pd

>>> model_name = "google/tapas-base"
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?


key: _node_content
string_value:
{"id_": "3feb354d-a3fa-48d8-91b5-f76e117d1fe1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35b5cb87-a786-4c60-82af-335a6bdf07a8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "197055e2896a9033362415f589ac2647bd715cce8e27b70f6190a9fc6a4da435", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cfc9bbf4-cb35-49bd-8102-70b6896cd8a3", "node_type": "1", "metadata": {}, "hash": "1aca70a288eb127f649e47744011c0fccf867966ce500164d4cab620c34e7516", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14919, "end_char_idx": 17968, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**STEP 3: Convert your data into tensors using TapasTokenizer**

<frameworkcontent>
<pt>
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different
inputs to be fine-tuned:

| **Task**                           | **Required inputs**                                                                                                 |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |
|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |
| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |

[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:

```py
>>> from transformers import TapasTokenizer
>>> import pandas as pd

>>> model_name = "google/tapas-base"
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table,
...     queries=queries,
...     answer_coordinates=answer_coordinates,
...     answer_text=answer_text,
...     padding="max_length",
...     return_tensors="pt",
... )
>>> inputs
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}
```

Note that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import torch
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


key: _node_content
string_value:
{"id_": "cfc9bbf4-cb35-49bd-8102-70b6896cd8a3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3feb354d-a3fa-48d8-91b5-f76e117d1fe1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "644bc8e515abf499a9c8ca27f59b15cd173f241d13ec900295ad21382409ea96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05d6bbe7-3387-4c08-be1c-ed57625792d4", "node_type": "1", "metadata": {}, "hash": "10ab27f0dffc801c0708d0f5be38c1d5b4d944fce26212363e9b01a37c3921c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16801, "end_char_idx": 19106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table,
...     queries=queries,
...     answer_coordinates=answer_coordinates,
...     answer_text=answer_text,
...     padding="max_length",
...     return_tensors="pt",
... )
>>> inputs
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}
```

Note that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import torch
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


>>> class TableDataset(torch.utils.data.Dataset):
...     def __init__(self, data, tokenizer):
...         self.data = data
...         self.tokenizer = tokenizer

...     def __getitem__(self, idx):
...         item = data.iloc[idx]
...         table = pd.read_csv(table_csv_path + item.table_file).astype(
...             str
...         )  # be sure to make your table data text only
...         encoding = self.tokenizer(
...             table=table,
...             queries=item.question,
...             answer_coordinates=item.answer_coordinates,
...             answer_text=item.answer_text,
...             truncation=True,
...             padding="max_length",
...             return_tensors="pt",
...         )
...         # remove the batch dimension which the tokenizer adds by default
...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}
...         # add the float_answer which is also required (weak supervision for aggregation case)
...         encoding["float_answer"] = torch.tensor(item.float_answer)
...         return encoding

...     def __len__(self):
...         return len(self.data)


key: _node_content
string_value:
{"id_": "05d6bbe7-3387-4c08-be1c-ed57625792d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cfc9bbf4-cb35-49bd-8102-70b6896cd8a3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "22befb726717d656ae180ac0772117d7bf38d69df35118d15f74fd8f5aea19eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "af66a608-9b0e-49b6-a33c-53304ba83a6c", "node_type": "1", "metadata": {}, "hash": "5490a13e6d08534c3f45d0017a5a95613556f89727d1492687d4ad4cac0f95ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17600, "end_char_idx": 22271, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


