# embedding_metadata

key: _node_content
string_value:
{"id_": "fa74fa73-374c-42c2-9fc6-21aecbbd31e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2e697c2e-2348-4f45-abff-028c40ffd859", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ee0351fb29b4ec6d83b8db7b79a8883f0da5f1bbd607d1f4315e299fb8e4877a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dad3e290-ac2a-456f-b4ae-9e8e27b438a6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5/", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "946879dfcd6633297ecae4220eddc6c26da67b72094b9bf22df1f6894e183805", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6523, "end_char_idx": 6669, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The dog is the one who does a ball in', ' in the park.']
```


## ByT5Tokenizer

[[autodoc]] ByT5Tokenizer

See [`ByT5Tokenizer`] for all details.


key: _node_content
string_value:
{"id_": "15c735c9-81d4-4ba8-aeae-67ee721c1884", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "039b025cc57a87ab4319c9d565047a9566dcf18b50319745188b575924840fef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44c58620-659a-4b4b-9651-680913fd6759", "node_type": "1", "metadata": {}, "hash": "6a9731a06952e7a5b0573498451d0fdf908e2a71fffc0d342a0e0be5c8ecf324", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SuperPoint

## Overview

The SuperPoint model was proposed
in [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629) by Daniel
DeTone, Tomasz Malisiewicz and Andrew Rabinovich.

This model is the result of a self-supervised training of a fully-convolutional network for interest point detection and
description. The model is able to detect interest points that are repeatable under homographic transformations and
provide a descriptor for each point. The use of the model in its own is limited, but it can be used as a feature
extractor for other tasks such as homography estimation, image matching, etc.

The abstract from the paper is the following:

*This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a
large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our
fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and
associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography
approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
when compared to LIFT, SIFT and ORB.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png"
alt="drawing" width="500"/>

<small> SuperPoint overview. Taken from the <a href="https://arxiv.org/abs/1712.07629v4">original paper.</a> </small>

## Usage tips

Here is a quick example of using the model to detect interest points in an image:

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
```

The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).

You can also feed multiple images to the model.


key: _node_content
string_value:
{"id_": "44c58620-659a-4b4b-9651-680913fd6759", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "039b025cc57a87ab4319c9d565047a9566dcf18b50319745188b575924840fef", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15c735c9-81d4-4ba8-aeae-67ee721c1884", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2113618f7ea83d440d2c2c2ea7704fc963e97810c349522d0649ba6c89696564", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1e935740-7796-4ada-9ed7-3f486b524779", "node_type": "1", "metadata": {}, "hash": "09da46a3b3df86abdc55d0ec6994a73466c32e30f9c1281aa502b712e3117a07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1100, "end_char_idx": 4342, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We introduce Homographic Adaptation, a multi-scale, multi-homography
approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
when compared to LIFT, SIFT and ORB.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png"
alt="drawing" width="500"/>

<small> SuperPoint overview. Taken from the <a href="https://arxiv.org/abs/1712.07629v4">original paper.</a> </small>

## Usage tips

Here is a quick example of using the model to detect interest points in an image:

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
```

The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).

You can also feed multiple images to the model. Due to the nature of SuperPoint, to output a dynamic number of keypoints,
you will need to use the mask attribute to retrieve the respective information :

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url_image_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
url_image_2 = "http://images.cocodataset.org/test-stuff2017/000000000568.jpg"
image_2 = Image.open(requests.get(url_image_2, stream=True).raw)

images = [image_1, image_2]

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(images, return_tensors="pt")
outputs = model(**inputs)

for i in range(len(images)):
    image_mask = outputs.mask[i]
    image_indices = torch.nonzero(image_mask).squeeze()
    image_keypoints = outputs.keypoints[i][image_indices]
    image_scores = outputs.scores[i][image_indices]
    image_descriptors = outputs.descriptors[i][image_indices]
```

You can then print the keypoints on the image to visualize the result :
```python
import cv2
for keypoint, score in zip(image_keypoints, image_scores):
    keypoint_x, keypoint_y = int(keypoint[0].item()), int(keypoint[1].item())
    color = tuple([score.item() * 255] * 3)
    image = cv2.circle(image, (keypoint_x, keypoint_y), 2, color)
cv2.imwrite("output_image.png", image)
```

This model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille).


key: _node_content
string_value:
{"id_": "1e935740-7796-4ada-9ed7-3f486b524779", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5d40ddbf-282b-4df1-a732-d9f4eab04aba", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "039b025cc57a87ab4319c9d565047a9566dcf18b50319745188b575924840fef", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44c58620-659a-4b4b-9651-680913fd6759", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint/", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6e818acac2aad1ccf43f3c3d9d4be2de9d605a61cc38e6490468dff76bc2be97", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4343, "end_char_idx": 5256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/magicleap/SuperPointPretrainedNetwork).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SuperPoint. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A notebook showcasing inference and visualization with SuperPoint can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SuperPoint/Inference_with_SuperPoint_to_detect_interest_points_in_an_image.ipynb). 🌎

## SuperPointConfig

[[autodoc]] SuperPointConfig

## SuperPointImageProcessor

[[autodoc]] SuperPointImageProcessor

- preprocess

## SuperPointForKeypointDetection

[[autodoc]] SuperPointForKeypointDetection

- forward


key: _node_content
string_value:
{"id_": "78677fcf-bea6-4465-bf29-9eed0fa09a03", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5/", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b8c4f7e3-032e-45c9-9e7f-91df001953d9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5/", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4f325eb26836e10fc74c745f1666dcd5bbab4c5fd2a76e6931e2b73c51a7d3b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1685, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FLAN-T5

## Overview

FLAN-T5 was released in the paper [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.

One can directly use FLAN-T5 weights without finetuning the model:

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
>>> tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")

>>> inputs = tokenizer("A step by step recipe to make bolognese pasta:", return_tensors="pt")
>>> outputs = model.generate(**inputs)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Pour a cup of bolognese into a large bowl and add the pasta']
```

FLAN-T5 includes the same improvements as T5 version 1.1 (see [here](https://huggingface.co/docs/transformers/model_doc/t5v1.1) for the full details of the model's improvements.)

Google has released the following variants:

- [google/flan-t5-small](https://huggingface.co/google/flan-t5-small)

- [google/flan-t5-base](https://huggingface.co/google/flan-t5-base)

- [google/flan-t5-large](https://huggingface.co/google/flan-t5-large)

- [google/flan-t5-xl](https://huggingface.co/google/flan-t5-xl)

- [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl).

The original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints).

<Tip>

Refer to [T5's documentation page](t5) for all API reference, code examples and notebooks. For more details regarding training and evaluation of the FLAN-T5, refer to the model card.

</Tip>


key: _node_content
string_value:
{"id_": "69c85a2c-6250-4d54-9acf-d6edbbe8baed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "98edeae81590169ba84bec73f31bdd7e0d7e418715887c74f87214fda04dc417", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a52b2faa-3fc9-4220-9003-968805848ec2", "node_type": "1", "metadata": {}, "hash": "38303f5ac0f88881c98e64927f80261931afd54ef45b4dae38fd068a8640a888", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2864, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CTRL

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=ctrl">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-ctrl-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/tiny-ctrl">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

CTRL model was proposed in [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus
of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).

The abstract from the paper is the following:

*Large-scale language models show promising text generation capabilities, but users cannot easily control particular
aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,
trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while
providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the
training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data
via model-based source attribution.*

This model was contributed by [keskarnitishr](https://huggingface.co/keskarnitishr). The original code can be found
[here](https://github.com/salesforce/ctrl).

## Usage tips

- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences
  or links to generate coherent text. Refer to the [original implementation](https://github.com/salesforce/ctrl) for
  more information.
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be
  observed in the *run_generation.py* example script.
- The PyTorch models can take the `past_key_values` as input, which is the previously computed key/value attention pairs.
  TensorFlow models accepts `past` as input. Using the `past_key_values` value prevents the model from re-computing
  pre-computed values in the context of text generation. See the [`forward`](model_doc/ctrl#transformers.CTRLModel.forward)
  method for more information on the usage of this argument.


key: _node_content
string_value:
{"id_": "a52b2faa-3fc9-4220-9003-968805848ec2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ad9229-4b27-4547-a68e-95bf3414bdd1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "98edeae81590169ba84bec73f31bdd7e0d7e418715887c74f87214fda04dc417", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "69c85a2c-6250-4d54-9acf-d6edbbe8baed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl/", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "34691692b54443ed90155d0ea3bc5676f14b8053a3d0503f9b19cf8cb03a0987", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2867, "end_char_idx": 3600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## CTRLConfig

[[autodoc]] CTRLConfig

## CTRLTokenizer

[[autodoc]] CTRLTokenizer
    - save_vocabulary

<frameworkcontent>
<pt>

## CTRLModel

[[autodoc]] CTRLModel
    - forward

## CTRLLMHeadModel

[[autodoc]] CTRLLMHeadModel
    - forward

## CTRLForSequenceClassification

[[autodoc]] CTRLForSequenceClassification
    - forward

</pt>
<tf>

## TFCTRLModel

[[autodoc]] TFCTRLModel
    - call

## TFCTRLLMHeadModel

[[autodoc]] TFCTRLLMHeadModel
    - call

## TFCTRLForSequenceClassification

[[autodoc]] TFCTRLForSequenceClassification
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "d4fe0adf-449d-4a55-abe6-e9eae7684b12", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/git/", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "728dd6d5-7c2b-48ad-b88f-7acedde5a47d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/git/", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9991feabf760a4a60cc6de008ed23be22b321f655538112c74dfd493ec6beb73", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3136, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GIT

## Overview

The GIT model was proposed in [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. GIT is a decoder-only Transformer
that leverages [CLIP](clip)'s vision encoder to condition the model on vision inputs besides text. The model obtains state-of-the-art results on
image captioning and visual question answering benchmarks.

The abstract from the paper is the following:

*In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/git_architecture.jpg"
alt="drawing" width="600"/>

<small> GIT architecture. Taken from the <a href="https://arxiv.org/abs/2205.14100" target="_blank">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).

## Usage tips

- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GIT.

- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).
- See also: [Causal language modeling task guide](../tasks/language_modeling)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.

## GitVisionConfig

[[autodoc]] GitVisionConfig

## GitVisionModel

[[autodoc]] GitVisionModel
    - forward

## GitConfig

[[autodoc]] GitConfig
    - all

## GitProcessor

[[autodoc]] GitProcessor
    - __call__

## GitModel

[[autodoc]] GitModel
    - forward

## GitForCausalLM

[[autodoc]] GitForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "31e7d217-5213-447b-bb49-63089328d058", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clap/", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "640990f0-1b8f-4f25-9444-e37d467ec908", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clap/", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1bc00e50582335e4f69118cb16cdc805a169326d826e17529a3c3ac861344f17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLAP

## Overview

The CLAP model was proposed in [Large Scale Contrastive Language-Audio pretraining with
feature fusion and keyword-to-caption augmentation](https://arxiv.org/pdf/2211.06687.pdf) by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.

CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.

The abstract from the paper is the following:

*Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-6*

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .
The original code can be found [here](https://github.com/LAION-AI/Clap).

## ClapConfig

[[autodoc]] ClapConfig
    - from_text_audio_configs

## ClapTextConfig

[[autodoc]] ClapTextConfig

## ClapAudioConfig

[[autodoc]] ClapAudioConfig

## ClapFeatureExtractor

[[autodoc]] ClapFeatureExtractor

## ClapProcessor

[[autodoc]] ClapProcessor

## ClapModel

[[autodoc]] ClapModel
    - forward
    - get_text_features
    - get_audio_features

## ClapTextModel

[[autodoc]] ClapTextModel
    - forward

## ClapTextModelWithProjection

[[autodoc]] ClapTextModelWithProjection
    - forward

## ClapAudioModel

[[autodoc]] ClapAudioModel
    - forward

## ClapAudioModelWithProjection

[[autodoc]] ClapAudioModelWithProjection
    - forward


key: _node_content
string_value:
{"id_": "ae6ef57d-9e12-4cf6-a137-cd9e724db0d7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "50aede24859417b94ac4baee45e0dc1414e7c2c3ce127d5086c85ff672e3255e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2156a421-3b4e-48a5-8627-dc6549d3b816", "node_type": "1", "metadata": {}, "hash": "10b1b79716168c7d477c667436e603efe4a3b6755e957eb60bc18aafe6014e1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLVP

## Overview

The CLVP (Contrastive Language-Voice Pretrained Transformer) model was proposed in [Better speech synthesis through scaling](https://arxiv.org/abs/2305.07243) by James Betker.

The abstract from the paper is the following:

*In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise - an expressive, multi-voice text-to-speech system.*


This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/neonbjb/tortoise-tts).


## Usage tips

1. CLVP is an integral part of the Tortoise TTS model.
2. CLVP can be used to compare different generated speech candidates with the provided text, and the best speech tokens are forwarded to the diffusion model.
3. The use of the [`ClvpModelForConditionalGeneration.generate()`] method is strongly recommended for tortoise usage.
4. Note that the CLVP model expects the audio to be sampled at 22.05 kHz contrary to other audio models which expects 16 kHz. 


## Brief Explanation:

- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.
- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.
- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.
- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. 
- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. 
- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  


Example :

```python
>>> import datasets
>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration

>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).
>>> text = "This is an example text."

>>> ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050))
>>> sample = ds[0]["audio"]

>>> # Define processor and model.
>>> processor = ClvpProcessor.from_pretrained("susnato/clvp_dev")
>>> model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev")

>>> # Generate processor output and model output.
>>> processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt")
>>> generated_output = model.generate(**processor_output)
```


key: _node_content
string_value:
{"id_": "2156a421-3b4e-48a5-8627-dc6549d3b816", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1a34160-fa37-49ce-8bd2-d5433b2e2520", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "50aede24859417b94ac4baee45e0dc1414e7c2c3ce127d5086c85ff672e3255e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae6ef57d-9e12-4cf6-a137-cd9e724db0d7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp/", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "939ae63c5cf46bbe4f9ff653c4c3a29697074cd2d91cedda40eed169be9c0882", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1436, "end_char_idx": 3958, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Brief Explanation:

- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.
- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.
- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.
- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. 
- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. 
- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  


Example :

```python
>>> import datasets
>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration

>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).
>>> text = "This is an example text."

>>> ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050))
>>> sample = ds[0]["audio"]

>>> # Define processor and model.
>>> processor = ClvpProcessor.from_pretrained("susnato/clvp_dev")
>>> model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev")

>>> # Generate processor output and model output.
>>> processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt")
>>> generated_output = model.generate(**processor_output)
```


## ClvpConfig

[[autodoc]] ClvpConfig
    - from_sub_model_configs

## ClvpEncoderConfig

[[autodoc]] ClvpEncoderConfig

## ClvpDecoderConfig

[[autodoc]] ClvpDecoderConfig

## ClvpTokenizer

[[autodoc]] ClvpTokenizer
    - save_vocabulary

## ClvpFeatureExtractor

[[autodoc]] ClvpFeatureExtractor
    - __call__

## ClvpProcessor

[[autodoc]] ClvpProcessor
    - __call__
    - decode
    - batch_decode

## ClvpModelForConditionalGeneration

[[autodoc]] ClvpModelForConditionalGeneration
    - forward
    - generate
    - get_text_features
    - get_speech_features

## ClvpForCausalLM

[[autodoc]] ClvpForCausalLM

## ClvpModel

[[autodoc]] ClvpModel

## ClvpEncoder

[[autodoc]] ClvpEncoder

## ClvpDecoder

[[autodoc]] ClvpDecoder


key: _node_content
string_value:
{"id_": "d2d23c93-155b-48f8-94b1-f9ef225b89ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma/", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f0fbb907-d2ef-4eff-898f-d1b2886e477d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma/", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f01fa09aca8de0f18304c2e8a6b3fd118660ee4f49d5b2e47e2afb0d82af8532", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RecurrentGemma

## Overview

The Recurrent Gemma model was proposed in [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://storage.googleapis.com/deepmind-media/gemma/recurrentgemma-report.pdf) by the Griffin, RLHF and Gemma Teams of Google.

The abstract from the paper is the following:

*We introduce RecurrentGemma, an open language model which uses Google’s novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.*

Tips:

- The original checkpoints can be converted using the conversion script [`src/transformers/models/recurrent_gemma/convert_recurrent_gemma_weights_to_hf.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/recurrent_gemma/convert_recurrent_gemma_to_hf.py). 

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The original code can be found [here](https://github.com/google-deepmind/recurrentgemma).


## RecurrentGemmaConfig

[[autodoc]] RecurrentGemmaConfig


## RecurrentGemmaModel

[[autodoc]] RecurrentGemmaModel
    - forward

## RecurrentGemmaForCausalLM

[[autodoc]] RecurrentGemmaForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "9846223e-4b23-478b-a3aa-c23901bac116", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe6066a3-1cea-462f-ba1d-7f2b772b55e8", "node_type": "1", "metadata": {}, "hash": "7cc0bb5b80177fc43726039da8bbe670c7e2fc7261cb0c4965eaced23a0ba463", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3289, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DETR

## Overview

The DETR model was proposed in [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. DETR
consists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for
object detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use
things like region proposals, non-maximum suppression procedure and anchor generation. Moreover, DETR can also be
naturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.

The abstract from the paper is the following:

*We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the
detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression
procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the
new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via
bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries,
DETR reasons about the relations of the objects and the global image context to directly output the final set of
predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many
other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and
highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily
generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive
baselines.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).

## How DETR works

Here's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:

First, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use
ResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a
tensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone
outputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is
then projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a
`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the
feature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =
`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually
longer than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).

Next, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider
these as image features).


key: _node_content
string_value:
{"id_": "fe6066a3-1cea-462f-ba1d-7f2b772b55e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9846223e-4b23-478b-a3aa-c23901bac116", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9862c18c83e422aa3cd14d1b205dc353eb833b18eae0beb2f98fa704b28a9b40", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "169a465d-4e47-4e76-9fab-72f0d13fab1c", "node_type": "1", "metadata": {}, "hash": "23941821e25ce2b1aa060ae85a540c4ab9aca2358db261fdbc09e5e92424a9fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1778, "end_char_idx": 5025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Moreover, DETR can be easily
generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive
baselines.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).

## How DETR works

Here's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:

First, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use
ResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a
tensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone
outputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is
then projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a
`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the
feature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =
`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually
longer than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).

Next, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider
these as image features). Next, so-called **object queries** are sent through the decoder. This is a tensor of shape
`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.
These input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to
the encoder, they are added to the input of each attention layer. Each object query will look for a particular object
in the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers
to output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads
are added on top for object detection: a linear layer for classifying each object query into one of the objects or "no
object", and a MLP to predict bounding boxes for each query.

The model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +
bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N
(so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as
bounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find
an optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for
the classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the
bounding boxes) are used to optimize the parameters of the model.

DETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance
segmentation).


key: _node_content
string_value:
{"id_": "169a465d-4e47-4e76-9fab-72f0d13fab1c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe6066a3-1cea-462f-ba1d-7f2b772b55e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c8ed1a285dcab7ed43636e6a91018b2714e23a26e9acc031303f0d7276949d2d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f", "node_type": "1", "metadata": {}, "hash": "981778274d11ee1aaa6676a47f03930acc7a7e47bf9b12c3828c00c246a9aa6f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3290, "end_char_idx": 6686, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next, so-called **object queries** are sent through the decoder. This is a tensor of shape
`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.
These input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to
the encoder, they are added to the input of each attention layer. Each object query will look for a particular object
in the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers
to output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads
are added on top for object detection: a linear layer for classifying each object query into one of the objects or "no
object", and a MLP to predict bounding boxes for each query.

The model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +
bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N
(so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as
bounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find
an optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for
the classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the
bounding boxes) are used to optimize the parameters of the model.

DETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance
segmentation). [`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of
[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,
where one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both
"things" (instances) and "stuff" (background things like trees, roads, sky), then freeze all the weights and train only
the mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is
required for the training to be possible, since the Hungarian matching is computed using distances between boxes.

## Usage tips

- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum
  number of objects that can be detected in a single image, and is set to 100 by default (see parameter
  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the
  authors used 100, while the maximum number of objects in a COCO image is ~70).
- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,
  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.
- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting
  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned
  absolute position embeddings. By default, the parameter `position_embedding_type` of
  [`~transformers.DetrConfig`] is set to `"sine"`.


key: _node_content
string_value:
{"id_": "8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "169a465d-4e47-4e76-9fab-72f0d13fab1c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3702a37c3f6cf4844a5e0a6c785b0a858d47dbaf0a6396232e9dacc9785159c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7402523-1d0a-4be4-99c6-ebce49150bab", "node_type": "1", "metadata": {}, "hash": "e8abbb78a57232433b5617e627f1746cdae35cb8de3362664cd2bfdd92f3f0bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5026, "end_char_idx": 8320, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of
[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,
where one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both
"things" (instances) and "stuff" (background things like trees, roads, sky), then freeze all the weights and train only
the mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is
required for the training to be possible, since the Hungarian matching is computed using distances between boxes.

## Usage tips

- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum
  number of objects that can be detected in a single image, and is set to 100 by default (see parameter
  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the
  authors used 100, while the maximum number of objects in a COCO image is ~70).
- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,
  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.
- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting
  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned
  absolute position embeddings. By default, the parameter `position_embedding_type` of
  [`~transformers.DetrConfig`] is set to `"sine"`.
- During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help
  the model output the correct number of objects of each class. If you set the parameter `auxiliary_loss` of
  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses
  are added after each decoder layer (with the FFNs sharing parameters).
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be
  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).
- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with
  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).
  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of
  [`~transformers.DetrConfig`] to `"tf_mobilenetv3_small_075"`, and then initializing the model with that
  config.
- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is
  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at
  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800.


key: _node_content
string_value:
{"id_": "e7402523-1d0a-4be4-99c6-ebce49150bab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e8d2ce8-7ff2-4cbc-8c59-2447ed395b8f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dafec49496f3a7a1ff2a3371b153c4f49b686f057c3b51118e0aa366b212fb5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59201cc9-7d7f-411a-bcdd-4dbe8d1eda45", "node_type": "1", "metadata": {}, "hash": "6659784809a4f14b87e33413e6ae2022780aaaae3758eabb9f9574ab3e1e0b60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6869, "end_char_idx": 10172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you set the parameter `auxiliary_loss` of
  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses
  are added after each decoder layer (with the FFNs sharing parameters).
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be
  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).
- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with
  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).
  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of
  [`~transformers.DetrConfig`] to `"tf_mobilenetv3_small_075"`, and then initializing the model with that
  config.
- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is
  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at
  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800. One can use
  [`~transformers.DetrImageProcessor`] to prepare images (and optional annotations in COCO format) for the
  model. Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the
  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.
  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using
  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].
- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.
  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.

There are three ways to instantiate a DETR model (depending on what you prefer):

Option 1: Instantiate DETR with pre-trained weights for entire model
```py
>>> from transformers import DetrForObjectDetection

>>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
```

Option 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone
```py
>>> from transformers import DetrConfig, DetrForObjectDetection

>>> config = DetrConfig()
>>> model = DetrForObjectDetection(config)
```
Option 3: Instantiate DETR with randomly initialized weights for backbone + Transformer
```py
>>> config = DetrConfig(use_pretrained_backbone=False)
>>> model = DetrForObjectDetection(config)
```

As a summary, consider the following table:

| Task | Object detection | Instance segmentation | Panoptic segmentation |
|------|------------------|-----------------------|-----------------------|
| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e.


key: _node_content
string_value:
{"id_": "59201cc9-7d7f-411a-bcdd-4dbe8d1eda45", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e7402523-1d0a-4be4-99c6-ebce49150bab", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "362717a3b94b0ea18caaa071a40544a8d38859e4e878009159ac29a92dc7f919", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f82c71aa-9290-4650-832d-756ef4e5e456", "node_type": "1", "metadata": {}, "hash": "c095f323596f810559cad014578152e749735cd92decc98e9f4cd0d7b7291f54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8449, "end_char_idx": 11155, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the
  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.
  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using
  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].
- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.
  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.

There are three ways to instantiate a DETR model (depending on what you prefer):

Option 1: Instantiate DETR with pre-trained weights for entire model
```py
>>> from transformers import DetrForObjectDetection

>>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
```

Option 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone
```py
>>> from transformers import DetrConfig, DetrForObjectDetection

>>> config = DetrConfig()
>>> model = DetrForObjectDetection(config)
```
Option 3: Instantiate DETR with randomly initialized weights for backbone + Transformer
```py
>>> config = DetrConfig(use_pretrained_backbone=False)
>>> model = DetrForObjectDetection(config)
```

As a summary, consider the following table:

| Task | Object detection | Instance segmentation | Panoptic segmentation |
|------|------------------|-----------------------|-----------------------|
| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e. instances) in an image | Predicting masks around both objects (i.e. instances) as well as "stuff" (i.e. background things like trees and roads) in an image |
| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |
| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |
| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |
| **Postprocessing** (i.e.


key: _node_content
string_value:
{"id_": "f82c71aa-9290-4650-832d-756ef4e5e456", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59201cc9-7d7f-411a-bcdd-4dbe8d1eda45", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3add9354cf0f8a9ed3d326240908388467b1be762fa047f8f5508bff1146f3fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "187612dd-2941-46b1-bc41-e060c3578239", "node_type": "1", "metadata": {}, "hash": "160e1d9ce59d6d183ea4fafeaca9384dca7efdad9bec24cfb6e4c439882240ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10173, "end_char_idx": 13285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
instances) in an image | Predicting masks around both objects (i.e. instances) as well as "stuff" (i.e. background things like trees and roads) in an image |
| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |
| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |
| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |
| **Postprocessing** (i.e. converting the output of the model to Pascal VOC format) | [`~transformers.DetrImageProcessor.post_process`] | [`~transformers.DetrImageProcessor.post_process_segmentation`] | [`~transformers.DetrImageProcessor.post_process_segmentation`], [`~transformers.DetrImageProcessor.post_process_panoptic`] |
| **evaluators** | `CocoEvaluator` with `iou_types="bbox"` | `CocoEvaluator` with `iou_types="bbox"` or `"segm"` | `CocoEvaluator` with `iou_tupes="bbox"` or `"segm"`, `PanopticEvaluator` |

In short, one should prepare the data either in COCO detection or COCO panoptic format, then use
[`~transformers.DetrImageProcessor`] to create `pixel_values`, `pixel_mask` and optional
`labels`, which can then be used to train (or fine-tune) a model. For evaluation, one should first convert the
outputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can
be be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like
mean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DETR.

<PipelineTag pipeline="object-detection"/>

- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).
- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!


key: _node_content
string_value:
{"id_": "187612dd-2941-46b1-bc41-e060c3578239", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ecda19b-c73b-45f4-a33c-29c52600449d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0dac43705e07d97e882b63b49d3cdd4663b5abf70534257a4dc77f60e93c2fa4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f82c71aa-9290-4650-832d-756ef4e5e456", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr/", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4f8ed5a8e6ba5bde625d23ed6f7673feb6512adeceb9e1d23c84da6ce20ae5e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11900, "end_char_idx": 14294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For evaluation, one should first convert the
outputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can
be be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like
mean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DETR.

<PipelineTag pipeline="object-detection"/>

- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).
- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DetrConfig

[[autodoc]] DetrConfig

## DetrImageProcessor

[[autodoc]] DetrImageProcessor
    - preprocess
    - post_process_object_detection
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## DetrFeatureExtractor

[[autodoc]] DetrFeatureExtractor
    - __call__
    - post_process_object_detection
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## DETR specific outputs

[[autodoc]] models.detr.modeling_detr.DetrModelOutput

[[autodoc]] models.detr.modeling_detr.DetrObjectDetectionOutput

[[autodoc]] models.detr.modeling_detr.DetrSegmentationOutput

## DetrModel

[[autodoc]] DetrModel
    - forward

## DetrForObjectDetection

[[autodoc]] DetrForObjectDetection
    - forward

## DetrForSegmentation

[[autodoc]] DetrForSegmentation
    - forward


key: _node_content
string_value:
{"id_": "814b4690-5a4e-4bf0-9e02-46e9386442bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1f09815cf50e614ea530014995f54175ff337f6d66bdff9a638df0ca1088d1da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e57bc7b-cda4-4171-8424-c262a0ae44f8", "node_type": "1", "metadata": {}, "hash": "6852a40e77a730b72069745d1c2dca914293210f9552d58125fa06d09af8b6de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLNet

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=xlnet">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlnet-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/xlnet-base-cased">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The XLNet model was proposed in [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov,
Quoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn
bidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization
order.

The abstract from the paper is the following:

*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves
better performance than pretraining approaches based on autoregressive language modeling. However, relying on
corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a
pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive
pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all
permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into
pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large
margin, including question answering, natural language inference, sentiment analysis, and document ranking.*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).

## Usage tips

- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained
  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.
- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
  `target_mapping` inputs to control the attention span and outputs (see examples in
  *examples/pytorch/text-generation/run_generation.py*)
- XLNet is one of the few models that has no sequence length limit.
- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,…,sequence length.
- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies.


key: _node_content
string_value:
{"id_": "5e57bc7b-cda4-4171-8424-c262a0ae44f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e66800f8-85ff-4e3f-91f0-74ee952b4c62", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1f09815cf50e614ea530014995f54175ff337f6d66bdff9a638df0ca1088d1da", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "814b4690-5a4e-4bf0-9e02-46e9386442bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet/", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2544b40c5569547b412b232afddea4f71f6b6b25ed62c3e9049a992317bced35", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1577, "end_char_idx": 5969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into
pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large
margin, including question answering, natural language inference, sentiment analysis, and document ranking.*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).

## Usage tips

- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained
  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.
- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
  `target_mapping` inputs to control the attention span and outputs (see examples in
  *examples/pytorch/text-generation/run_generation.py*)
- XLNet is one of the few models that has no sequence length limit.
- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,…,sequence length.
- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## XLNetConfig

[[autodoc]] XLNetConfig

## XLNetTokenizer

[[autodoc]] XLNetTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## XLNetTokenizerFast

[[autodoc]] XLNetTokenizerFast

## XLNet specific outputs

[[autodoc]] models.xlnet.modeling_xlnet.XLNetModelOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForMultipleChoiceOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForTokenClassificationOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput

<frameworkcontent>
<pt>

## XLNetModel

[[autodoc]] XLNetModel
    - forward

## XLNetLMHeadModel

[[autodoc]] XLNetLMHeadModel
    - forward

## XLNetForSequenceClassification

[[autodoc]] XLNetForSequenceClassification
    - forward

## XLNetForMultipleChoice

[[autodoc]] XLNetForMultipleChoice
    - forward

## XLNetForTokenClassification

[[autodoc]] XLNetForTokenClassification
    - forward

## XLNetForQuestionAnsweringSimple

[[autodoc]] XLNetForQuestionAnsweringSimple
    - forward

## XLNetForQuestionAnswering

[[autodoc]] XLNetForQuestionAnswering
    - forward

</pt>
<tf>

## TFXLNetModel

[[autodoc]] TFXLNetModel
    - call

## TFXLNetLMHeadModel

[[autodoc]] TFXLNetLMHeadModel
    - call

## TFXLNetForSequenceClassification

[[autodoc]] TFXLNetForSequenceClassification
    - call

## TFLNetForMultipleChoice

[[autodoc]] TFXLNetForMultipleChoice
    - call

## TFXLNetForTokenClassification

[[autodoc]] TFXLNetForTokenClassification
    - call

## TFXLNetForQuestionAnsweringSimple

[[autodoc]] TFXLNetForQuestionAnsweringSimple
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "78e43ddd-3213-4268-a5f6-ccb808e3cf43", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7d65f87bda1d85b5f7510d98b948f32ec8528c9d507578cdac6cca58ba536b45", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50d7ceed-a11d-4bf9-86df-39c236274b6a", "node_type": "1", "metadata": {}, "hash": "0b564a1142000f05d18fb6d5c78e5f38ee094ba146a5af9cefc2af2d15d4a299", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3707, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ESM

## Overview

This page provides code and pre-trained weights for Transformer protein language models from Meta AI's Fundamental 
AI Research Team, providing the state-of-the-art ESMFold and ESM-2, and the previously released ESM-1b and ESM-1v.
Transformer protein language models were introduced in the paper [Biological structure and function emerge from scaling
unsupervised learning to 250 million protein sequences](https://www.pnas.org/content/118/15/e2016239118) by 
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, 
C. Lawrence Zitnick, Jerry Ma, and Rob Fergus.
The first version of this paper was [preprinted in 2019](https://www.biorxiv.org/content/10.1101/622803v1?versioned=true).

ESM-2 outperforms all tested single-sequence protein language models across a range of structure prediction tasks,
and enables atomic resolution structure prediction.
It was released with the paper [Language models of protein sequences at the scale of evolution enable accurate
structure prediction](https://doi.org/10.1101/2022.07.20.500902) by Zeming Lin, Halil Akin, Roshan Rao, Brian Hie,
Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido and Alexander Rives.

Also introduced in this paper was ESMFold. It uses an ESM-2 stem with a head that can predict folded protein
structures with state-of-the-art accuracy. Unlike [AlphaFold2](https://www.nature.com/articles/s41586-021-03819-2),
it relies on the token embeddings from the large pre-trained protein language model stem and does not perform a multiple
sequence alignment (MSA) step at inference time, which means that ESMFold checkpoints are fully "standalone" -
they do not require a database of known protein sequences and structures with associated external query tools
to make predictions, and are much faster as a result.


The abstract from 
"Biological structure and function emerge from scaling unsupervised learning to 250 
million protein sequences" is


*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised
learning has led to major advances in representation learning and statistical generation. In the life sciences, the
anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling
at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To
this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250
million protein sequences spanning evolutionary diversity. The resulting model contains information about biological
properties in its representations. The representations are learned from sequence data alone. The learned representation
space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to
remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and
can be identified by linear projections. Representation learning produces features that generalize across a range of
applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and
improving state-of-the-art features for long-range contact prediction.*


The abstract from
"Language models of protein sequences at the scale of evolution enable accurate structure prediction" is

*Large language models have recently been shown to develop emergent capabilities with scale, going beyond
simple pattern matching to perform higher level reasoning and generate lifelike images and text.


key: _node_content
string_value:
{"id_": "50d7ceed-a11d-4bf9-86df-39c236274b6a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7d65f87bda1d85b5f7510d98b948f32ec8528c9d507578cdac6cca58ba536b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78e43ddd-3213-4268-a5f6-ccb808e3cf43", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "be875441324158dd07e4336889ae7fec948a6e9dd6554e66ec17f800071e45d0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "424e32cc-d6f4-47a8-bb30-1fd361c987a9", "node_type": "1", "metadata": {}, "hash": "fa43d8f6effcc0541c60b7e2482ddef4cc2520021fc3f1ee6d76d3710289f825", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1887, "end_char_idx": 5415, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract from 
"Biological structure and function emerge from scaling unsupervised learning to 250 
million protein sequences" is


*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised
learning has led to major advances in representation learning and statistical generation. In the life sciences, the
anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling
at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To
this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250
million protein sequences spanning evolutionary diversity. The resulting model contains information about biological
properties in its representations. The representations are learned from sequence data alone. The learned representation
space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to
remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and
can be identified by linear projections. Representation learning produces features that generalize across a range of
applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and
improving state-of-the-art features for long-range contact prediction.*


The abstract from
"Language models of protein sequences at the scale of evolution enable accurate structure prediction" is

*Large language models have recently been shown to develop emergent capabilities with scale, going beyond
simple pattern matching to perform higher level reasoning and generate lifelike images and text. While
language models trained on protein sequences have been studied at a smaller scale, little is known about
what they learn about biology as they are scaled up. In this work we train models up to 15 billion parameters,
the largest language models of proteins to be evaluated to date. We find that as models are scaled they learn
information enabling the prediction of the three-dimensional structure of a protein at the resolution of
individual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly
from the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for
sequences with low perplexity that are well understood by the language model. ESMFold inference is an
order of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic
proteins in practical timescales.*

The original code can be found [here](https://github.com/facebookresearch/esm) and was
was developed by the Fundamental AI Research team at Meta AI.
ESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)
and [Matt](https://huggingface.co/Rocketknight1).

ESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and
[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their
help throughout the process!

## Usage tips

- ESM models are trained with a masked language modeling (MLM) objective.
- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0.


key: _node_content
string_value:
{"id_": "424e32cc-d6f4-47a8-bb30-1fd361c987a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1e0eb0e7-a29c-45eb-bf9d-ddc680960939", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7d65f87bda1d85b5f7510d98b948f32ec8528c9d507578cdac6cca58ba536b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50d7ceed-a11d-4bf9-86df-39c236274b6a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm/", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4b20c6a3ecfce7789566b0d16d68dd532e97c4c67b2e97eb6ef50511d1065ce5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3872, "end_char_idx": 6570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this work we train models up to 15 billion parameters,
the largest language models of proteins to be evaluated to date. We find that as models are scaled they learn
information enabling the prediction of the three-dimensional structure of a protein at the resolution of
individual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly
from the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for
sequences with low perplexity that are well understood by the language model. ESMFold inference is an
order of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic
proteins in practical timescales.*

The original code can be found [here](https://github.com/facebookresearch/esm) and was
was developed by the Fundamental AI Research team at Meta AI.
ESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)
and [Matt](https://huggingface.co/Rocketknight1).

ESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and
[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their
help throughout the process!

## Usage tips

- ESM models are trained with a masked language modeling (MLM) objective.
- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Masked language modeling task guide](../tasks/masked_language_modeling)

## EsmConfig

[[autodoc]] EsmConfig
    - all

## EsmTokenizer

[[autodoc]] EsmTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

<frameworkcontent>
<pt>

## EsmModel

[[autodoc]] EsmModel
    - forward

## EsmForMaskedLM

[[autodoc]] EsmForMaskedLM
    - forward

## EsmForSequenceClassification

[[autodoc]] EsmForSequenceClassification
    - forward

## EsmForTokenClassification

[[autodoc]] EsmForTokenClassification
    - forward

## EsmForProteinFolding

[[autodoc]] EsmForProteinFolding
    - forward

</pt>
<tf>

## TFEsmModel

[[autodoc]] TFEsmModel
    - call

## TFEsmForMaskedLM

[[autodoc]] TFEsmForMaskedLM
    - call

## TFEsmForSequenceClassification

[[autodoc]] TFEsmForSequenceClassification
    - call

## TFEsmForTokenClassification

[[autodoc]] TFEsmForTokenClassification
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "5d1a149d-f7b4-46ce-bb9d-ec2be55e53b3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt/", "title": "Pyramid Vision Transformer (PVT)", "tokens": 892, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5cc2cc-1b85-487b-9923-99e86f38f3fe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt/", "title": "Pyramid Vision Transformer (PVT)", "tokens": 892, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6beaae0361dffd95162d536c3d702cc441189a1b0ffe0a851c700d35f2b3fa43", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Pyramid Vision Transformer (PVT)

## Overview

The PVT model was proposed in
[Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://arxiv.org/abs/2102.12122)
by Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao. The PVT is a type of
vision transformer that utilizes a pyramid structure to make it an effective backbone for dense prediction tasks. Specifically
it allows for more fine-grained inputs (4 x 4 pixels per patch) to be used, while simultaneously shrinking the sequence length
of the Transformer as it deepens - reducing the computational cost. Additionally, a spatial-reduction attention (SRA) layer
is used to further reduce the resource consumption when learning high-resolution features.

The abstract from the paper is the following:

*Although convolutional neural networks (CNNs) have achieved great success in computer vision, this work investigates a 
simpler, convolution-free backbone network useful for many dense prediction tasks. Unlike the recently proposed Vision 
Transformer (ViT) that was designed for image classification specifically, we introduce the Pyramid Vision Transformer 
(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several 
merits compared to current state of the arts. Different from ViT that typically yields low resolution outputs and 
incurs high computational and memory costs, PVT not only can be trained on dense partitions of an image to achieve high 
output resolution, which is important for dense prediction, but also uses a progressive shrinking pyramid to reduce the 
computations of large feature maps. PVT inherits the advantages of both CNN and Transformer, making it a unified 
backbone for various vision tasks without convolutions, where it can be used as a direct replacement for CNN backbones. 
We validate PVT through extensive experiments, showing that it boosts the performance of many downstream tasks, including
object detection, instance and semantic segmentation. For example, with a comparable number of parameters, PVT+RetinaNet 
achieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute AP (see Figure 2). We hope 
that PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future research.*

This model was contributed by [Xrenya](https://huggingface.co/Xrenya). The original code can be found [here](https://github.com/whai362/PVT).


- PVTv1 on ImageNet-1K

| **Model variant**  |**Size** |**Acc@1**|**Params (M)**|
|--------------------|:-------:|:-------:|:------------:|
| PVT-Tiny           |    224  |   75.1  |     13.2     |
| PVT-Small          |    224  |   79.8  |     24.5     |
| PVT-Medium         |    224  |   81.2  |     44.2     |
| PVT-Large          |    224  |   81.7  |     61.4     |


## PvtConfig

[[autodoc]] PvtConfig

## PvtImageProcessor

[[autodoc]] PvtImageProcessor
    - preprocess

## PvtForImageClassification

[[autodoc]] PvtForImageClassification
    - forward

## PvtModel

[[autodoc]] PvtModel
    - forward


key: _node_content
string_value:
{"id_": "8311b772-6d45-4431-88e9-f67d1674d72a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "98a10bd3-13e6-41a2-8591-c8c2652170e7", "node_type": "1", "metadata": {}, "hash": "d4f73b6a6ca8e6f396f61c8ee559a3b59dcc31c8f41adc78ed9c513735185d76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3102, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Reformer

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=reformer">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-reformer-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/reformer-crime-and-punishment">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The Reformer model was proposed in the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451.pdf) by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.

The abstract from the paper is the following:

*Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can
be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of
Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its
complexity from O(L^2) to O(Llog(L)), where L is the length of the sequence. Furthermore, we use reversible residual
layers instead of the standard residuals, which allows storing activations only once in the training process instead of
N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models
while being much more memory-efficient and much faster on long sequences.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be
found [here](https://github.com/google/trax/tree/master/trax/models/reformer).

## Usage tips

- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).
- Use Axial position encoding (see below for more details). It’s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.
- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It’s a technique to avoid computing the full product query-key in the attention layers.
- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).
- Compute the feedforward operations by chunks and not on the whole batch.

### Axial Positional Encodings

Axial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embeddings vector of size \\(d\\) being the `config.hidden_size` for
every position \\(i, \ldots, n_s\\), with \\(n_s\\) being `config.max_embedding_size`.


key: _node_content
string_value:
{"id_": "98a10bd3-13e6-41a2-8591-c8c2652170e7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8311b772-6d45-4431-88e9-f67d1674d72a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6a7447e86e266685db428201b5966ed90854aeff04340cfe3af238313aeb9e35", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967", "node_type": "1", "metadata": {}, "hash": "0f69baf10066f574a8b8d309e1fb789ea227ec2930c8111245a30fb1f3c43cc7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1493, "end_char_idx": 3435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Authors' code can be
found [here](https://github.com/google/trax/tree/master/trax/models/reformer).

## Usage tips

- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).
- Use Axial position encoding (see below for more details). It’s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.
- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It’s a technique to avoid computing the full product query-key in the attention layers.
- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).
- Compute the feedforward operations by chunks and not on the whole batch.

### Axial Positional Encodings

Axial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embeddings vector of size \\(d\\) being the `config.hidden_size` for
every position \\(i, \ldots, n_s\\), with \\(n_s\\) being `config.max_embedding_size`. This means that having
a sequence length of \\(n_s = 2^{19} \approx 0.5M\\) and a `config.hidden_size` of \\(d = 2^{10} \approx 1000\\)
would result in a position encoding matrix:

$$X_{i,j}, \text{ with } i \in \left[1,\ldots, d\right] \text{ and } j \in \left[1,\ldots, n_s\right]$$

which alone has over 500M parameters to store.


key: _node_content
string_value:
{"id_": "dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98a10bd3-13e6-41a2-8591-c8c2652170e7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fba1c48eccae8338d2ba8321ded0b51751c287f31048c6e8ae9efea80c52e289", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b65c7299-2bc9-4593-a5a8-bd3b03008ac3", "node_type": "1", "metadata": {}, "hash": "37a387a0577b18f4a59a1f963a979ff706be00179cde55bf3c62d3d399429cd7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1972, "end_char_idx": 4393, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It’s a technique to avoid computing the full product query-key in the attention layers.
- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).
- Compute the feedforward operations by chunks and not on the whole batch.

### Axial Positional Encodings

Axial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embeddings vector of size \\(d\\) being the `config.hidden_size` for
every position \\(i, \ldots, n_s\\), with \\(n_s\\) being `config.max_embedding_size`. This means that having
a sequence length of \\(n_s = 2^{19} \approx 0.5M\\) and a `config.hidden_size` of \\(d = 2^{10} \approx 1000\\)
would result in a position encoding matrix:

$$X_{i,j}, \text{ with } i \in \left[1,\ldots, d\right] \text{ and } j \in \left[1,\ldots, n_s\right]$$

which alone has over 500M parameters to store. Axial positional encodings factorize \\(X_{i,j}\\) into two matrices:

$$X^{1}_{i,j}, \text{ with } i \in \left[1,\ldots, d^1\right] \text{ and } j \in \left[1,\ldots, n_s^1\right]$$

and

$$X^{2}_{i,j}, \text{ with } i \in \left[1,\ldots, d^2\right] \text{ and } j \in \left[1,\ldots, n_s^2\right]$$

with:

$$d = d^1 + d^2 \text{ and } n_s = n_s^1 \times n_s^2 .$$

Therefore the following holds:

$$X_{i,j} = \begin{cases}
X^{1}_{i, k}, & \text{if }\ i < d^1 \text{ with } k = j \mod n_s^1 \\
X^{2}_{i - d^1, l}, & \text{if } i \ge d^1 \text{ with } l = \lfloor\frac{j}{n_s^1}\rfloor
\end{cases}$$

Intuitively, this means that a position embedding vector \\(x_j \in \mathbb{R}^{d}\\) is now the composition of two
factorized embedding vectors: \\(x^1_{k, l} + x^2_{l, k}\\), where as the `config.max_embedding_size` dimension
\\(j\\) is factorized into \\(k \text{ and } l\\). This design ensures that each position embedding vector
\\(x_j\\) is unique.


key: _node_content
string_value:
{"id_": "b65c7299-2bc9-4593-a5a8-bd3b03008ac3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dab8ebd5-ee9f-446f-bf4e-6aa57e5cc967", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e6e57e7fddda71de401387e91e564729b246ddf7f9e26464662a5893bf3b145e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7366a839-9fbb-45ad-8d4e-f76d988cca48", "node_type": "1", "metadata": {}, "hash": "ab830199aa7d0cc38601c1afdbf9c4b39534517ae9843d8fd974688c695455d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3436, "end_char_idx": 5018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Axial positional encodings factorize \\(X_{i,j}\\) into two matrices:

$$X^{1}_{i,j}, \text{ with } i \in \left[1,\ldots, d^1\right] \text{ and } j \in \left[1,\ldots, n_s^1\right]$$

and

$$X^{2}_{i,j}, \text{ with } i \in \left[1,\ldots, d^2\right] \text{ and } j \in \left[1,\ldots, n_s^2\right]$$

with:

$$d = d^1 + d^2 \text{ and } n_s = n_s^1 \times n_s^2 .$$

Therefore the following holds:

$$X_{i,j} = \begin{cases}
X^{1}_{i, k}, & \text{if }\ i < d^1 \text{ with } k = j \mod n_s^1 \\
X^{2}_{i - d^1, l}, & \text{if } i \ge d^1 \text{ with } l = \lfloor\frac{j}{n_s^1}\rfloor
\end{cases}$$

Intuitively, this means that a position embedding vector \\(x_j \in \mathbb{R}^{d}\\) is now the composition of two
factorized embedding vectors: \\(x^1_{k, l} + x^2_{l, k}\\), where as the `config.max_embedding_size` dimension
\\(j\\) is factorized into \\(k \text{ and } l\\). This design ensures that each position embedding vector
\\(x_j\\) is unique.

Using the above example again, axial position encoding with \\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\)
can drastically reduced the number of parameters from 500 000 000 to \\(2^{18} + 2^{19} \approx 780 000\\) parameters, this means 85% less memory usage.

In practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\((d^1, d^2)\\) which sum has to be
equal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\((n_s^1, n_s^2)\\) which
product has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence
length* of the `input_ids`.


key: _node_content
string_value:
{"id_": "7366a839-9fbb-45ad-8d4e-f76d988cca48", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b65c7299-2bc9-4593-a5a8-bd3b03008ac3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9e1d1dd03e40f3503fbb0783f337de2ed711c2f3801a52ef8b5934134b239c88", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "02bcef94-aecd-4870-87cf-2d662e037754", "node_type": "1", "metadata": {}, "hash": "396889b2d203e7e020062d3dd57b0db89efcffcab854cdbf59464f8e729464b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4317, "end_char_idx": 7468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This design ensures that each position embedding vector
\\(x_j\\) is unique.

Using the above example again, axial position encoding with \\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\)
can drastically reduced the number of parameters from 500 000 000 to \\(2^{18} + 2^{19} \approx 780 000\\) parameters, this means 85% less memory usage.

In practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\((d^1, d^2)\\) which sum has to be
equal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\((n_s^1, n_s^2)\\) which
product has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence
length* of the `input_ids`.


### LSH Self Attention

In Locality sensitive hashing (LSH) self attention the key and query projection weights are tied. Therefore, the key
query embedding vectors are also tied. LSH self attention uses the locality sensitive hashing mechanism proposed in
[Practical and Optimal LSH for Angular Distance](https://arxiv.org/abs/1509.02897) to assign each of the tied key
query embedding vectors to one of `config.num_buckets` possible buckets. The premise is that the more "similar"
key query embedding vectors (in terms of *cosine similarity*) are to each other, the more likely they are assigned to
the same bucket.

The accuracy of the LSH mechanism can be improved by increasing `config.num_hashes` or directly the argument
`num_hashes` of the forward function so that the output of the LSH self attention better approximates the output
of the "normal" full self attention. The buckets are then sorted and chunked into query key embedding vector chunks
each of length `config.lsh_chunk_length`. For each chunk, the query embedding vectors attend to its key vectors
(which are tied to themselves) and to the key embedding vectors of `config.lsh_num_chunks_before` previous
neighboring chunks and `config.lsh_num_chunks_after` following neighboring chunks.

For more information, see the [original Paper](https://arxiv.org/abs/2001.04451) or this great [blog post](https://www.pragmatic.ml/reformer-deep-dive/).

Note that `config.num_buckets` can also be factorized into a list \\((n_{\text{buckets}}^1,
n_{\text{buckets}}^2)\\). This way instead of assigning the query key embedding vectors to one of \\((1,\ldots,
n_{\text{buckets}})\\) they are assigned to one of \\((1-1,\ldots, n_{\text{buckets}}^1-1, \ldots,
1-n_{\text{buckets}}^2, \ldots, n_{\text{buckets}}^1-n_{\text{buckets}}^2)\\). This is crucial for very long sequences to
save memory.

When training a model from scratch, it is recommended to leave `config.num_buckets=None`, so that depending on the
sequence length a good value for `num_buckets` is calculated on the fly. This value will then automatically be
saved in the config and should be reused for inference.

Using LSH self attention, the memory and time complexity of the query-key matmul operation can be reduced from
\\(\mathcal{O}(n_s \times n_s)\\) to \\(\mathcal{O}(n_s \times \log(n_s))\\), which usually represents the memory
and time bottleneck in a transformer model, with \\(n_s\\) being the sequence length.


key: _node_content
string_value:
{"id_": "02bcef94-aecd-4870-87cf-2d662e037754", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ed314aa-ddb6-4e35-a41c-d727ec9a3df0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "beebe666ab4868fa8004f081a546c629cc8f0e98045b4dfbda874d5b99b0ee5f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7366a839-9fbb-45ad-8d4e-f76d988cca48", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer/", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b5644b8499d1ad7868c789a8daa60571a09a3c954084e11541ced90919ce1a7d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7471, "end_char_idx": 9746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Local Self Attention

Local self attention is essentially a "normal" self attention layer with key, query and value projections, but is
chunked so that in each chunk of length `config.local_chunk_length` the query embedding vectors only attends to
the key embedding vectors in its chunk and to the key embedding vectors of `config.local_num_chunks_before`
previous neighboring chunks and `config.local_num_chunks_after` following neighboring chunks.

Using Local self attention, the memory and time complexity of the query-key matmul operation can be reduced from
\\(\mathcal{O}(n_s \times n_s)\\) to \\(\mathcal{O}(n_s \times \log(n_s))\\), which usually represents the memory
and time bottleneck in a transformer model, with \\(n_s\\) being the sequence length.


### Training

During training, we must ensure that the sequence length is set to a value that can be divided by the least common
multiple of `config.lsh_chunk_length` and `config.local_chunk_length` and that the parameters of the Axial
Positional Encodings are correctly set as described above. Reformer is very memory efficient so that the model can
easily be trained on sequences as long as 64000 tokens.

For training, the [`ReformerModelWithLMHead`] should be used as follows:

```python
input_ids = tokenizer.encode("This is a sentence from the training data", return_tensors="pt")
loss = model(input_ids, labels=input_ids)[0]
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)

## ReformerConfig

[[autodoc]] ReformerConfig

## ReformerTokenizer

[[autodoc]] ReformerTokenizer
    - save_vocabulary

## ReformerTokenizerFast

[[autodoc]] ReformerTokenizerFast

## ReformerModel

[[autodoc]] ReformerModel
    - forward

## ReformerModelWithLMHead

[[autodoc]] ReformerModelWithLMHead
    - forward

## ReformerForMaskedLM

[[autodoc]] ReformerForMaskedLM
    - forward

## ReformerForSequenceClassification

[[autodoc]] ReformerForSequenceClassification
    - forward

## ReformerForQuestionAnswering

[[autodoc]] ReformerForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "38124589-3776-42df-b021-6b73f9ef71d3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1f44f127-133b-4286-8ea0-29818831ad85", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c2e40960512fb3049555ca0526e4d4259be0943f11e3ed65756e6284e6c969f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12782af6-dd35-4994-82bc-aa0462797469", "node_type": "1", "metadata": {}, "hash": "6fe06c8a56f87437b587d671e99662b382783d5338bc58efe976412b0119f52b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1812, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CamemBERT

## Overview

The CamemBERT model was proposed in [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by
[Louis Martin](https://huggingface.co/louismartin), [Benjamin Muller](https://huggingface.co/benjamin-mlr), [Pedro Javier Ortiz Suárez](https://huggingface.co/pjox), Yoann Dupont, Laurent Romary, Éric Villemonte de la
Clergerie, [Djamé Seddah](https://huggingface.co/Djame), and [Benoît Sagot](https://huggingface.co/sagot). It is based on Facebook's RoBERTa model released in 2019. It is a model
trained on 138GB of French text.

The abstract from the paper is the following:

*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available
models have either been trained on English data or on the concatenation of data in multiple languages. This makes
practical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,
we release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the
performance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,
dependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art
for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and
downstream applications for French NLP.*

This model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).

<Tip>

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well 
as the information relative to the inputs and outputs.


key: _node_content
string_value:
{"id_": "12782af6-dd35-4994-82bc-aa0462797469", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1f44f127-133b-4286-8ea0-29818831ad85", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c2e40960512fb3049555ca0526e4d4259be0943f11e3ed65756e6284e6c969f4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "38124589-3776-42df-b021-6b73f9ef71d3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert/", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ddf6810e480d24013f1b84839ac5e493c46c3e2f4c45e1d9d310cd09552b905e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 467, "end_char_idx": 3589, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is based on Facebook's RoBERTa model released in 2019. It is a model
trained on 138GB of French text.

The abstract from the paper is the following:

*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available
models have either been trained on English data or on the concatenation of data in multiple languages. This makes
practical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,
we release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the
performance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,
dependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art
for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and
downstream applications for French NLP.*

This model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).

<Tip>

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well 
as the information relative to the inputs and outputs.

</Tip>

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## CamembertConfig

[[autodoc]] CamembertConfig

## CamembertTokenizer

[[autodoc]] CamembertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## CamembertTokenizerFast

[[autodoc]] CamembertTokenizerFast

<frameworkcontent>
<pt>

## CamembertModel

[[autodoc]] CamembertModel

## CamembertForCausalLM

[[autodoc]] CamembertForCausalLM

## CamembertForMaskedLM

[[autodoc]] CamembertForMaskedLM

## CamembertForSequenceClassification

[[autodoc]] CamembertForSequenceClassification

## CamembertForMultipleChoice

[[autodoc]] CamembertForMultipleChoice

## CamembertForTokenClassification

[[autodoc]] CamembertForTokenClassification

## CamembertForQuestionAnswering

[[autodoc]] CamembertForQuestionAnswering

</pt>
<tf>

## TFCamembertModel

[[autodoc]] TFCamembertModel

## TFCamembertForCasualLM

[[autodoc]] TFCamembertForCausalLM

## TFCamembertForMaskedLM

[[autodoc]] TFCamembertForMaskedLM

## TFCamembertForSequenceClassification

[[autodoc]] TFCamembertForSequenceClassification

## TFCamembertForMultipleChoice

[[autodoc]] TFCamembertForMultipleChoice

## TFCamembertForTokenClassification

[[autodoc]] TFCamembertForTokenClassification

## TFCamembertForQuestionAnswering

[[autodoc]] TFCamembertForQuestionAnswering

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "7d76109f-258d-4fd5-aa1a-b750981db8a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese/", "title": "GPT-NeoX-Japanese", "tokens": 867, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c921b33f-8b74-4e3d-abaf-c67a354f4556", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese/", "title": "GPT-NeoX-Japanese", "tokens": 867, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1f55da0e00641be0817d4a1e6c6b9cf607934c4643f66a10f342dca9d00b28b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPT-NeoX-Japanese

## Overview

We introduce GPT-NeoX-Japanese, which is an autoregressive language model for Japanese, trained on top of [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).
Japanese is a unique language with its large vocabulary and a combination of hiragana, katakana, and kanji writing scripts.
To address this distinct structure of the Japanese language, we use a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). We are very grateful to *tanreinama* for open-sourcing this incredibly helpful tokenizer.
Following the recommendations from Google's research on [PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we have removed bias parameters from transformer blocks, achieving better model performance. Please refer [this article](https://medium.com/ml-abeja/training-a-better-gpt-2-93b157662ae4) in detail.

Development of the model was led by [Shinya Otani](https://github.com/SO0529), [Takayoshi Makabe](https://github.com/spider-man-tm), [Anuj Arora](https://github.com/Anuj040), and [Kyo Hattori](https://github.com/go5paopao) from [ABEJA, Inc.](https://www.abejainc.com/). For more information on this model-building activity, please refer [here (ja)](https://tech-blog.abeja.asia/entry/abeja-gpt-project-202207).

### Usage example

The `generate()` method can be used to generate text using GPT NeoX Japanese model.

```python
>>> from transformers import GPTNeoXJapaneseForCausalLM, GPTNeoXJapaneseTokenizer

>>> model = GPTNeoXJapaneseForCausalLM.from_pretrained("abeja/gpt-neox-japanese-2.7b")
>>> tokenizer = GPTNeoXJapaneseTokenizer.from_pretrained("abeja/gpt-neox-japanese-2.7b")

>>> prompt = "人とAIが協調するためには、"

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]

>>> print(gen_text)
人とAIが協調するためには、AIと人が共存し、AIを正しく理解する必要があります。
```

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## GPTNeoXJapaneseConfig

[[autodoc]] GPTNeoXJapaneseConfig

## GPTNeoXJapaneseTokenizer

[[autodoc]] GPTNeoXJapaneseTokenizer

## GPTNeoXJapaneseModel

[[autodoc]] GPTNeoXJapaneseModel
    - forward

## GPTNeoXJapaneseForCausalLM

[[autodoc]] GPTNeoXJapaneseForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "714f9684-ca93-421f-8fd7-77e268243a92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mra/", "title": "MRA", "tokens": 618, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb199c94-1e71-4016-a670-f6e70f08b4d5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mra/", "title": "MRA", "tokens": 618, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ac3499e13f261e5b192296dc634f7b2aa5c4c56cef782f9348b0c29f315218f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2004, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MRA

## Overview

The MRA model was proposed in [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://arxiv.org/abs/2207.10284) by Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh.

The abstract from the paper is the following:

*Transformers have emerged as a preferred model for many tasks in natural language processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at https://github.com/mlpen/mra-attention.*

This model was contributed by [novice03](https://huggingface.co/novice03).
The original code can be found [here](https://github.com/mlpen/mra-attention).

## MraConfig

[[autodoc]] MraConfig

## MraModel

[[autodoc]] MraModel
    - forward

## MraForMaskedLM

[[autodoc]] MraForMaskedLM
    - forward

## MraForSequenceClassification

[[autodoc]] MraForSequenceClassification
    - forward

## MraForMultipleChoice

[[autodoc]] MraForMultipleChoice
    - forward

## MraForTokenClassification

[[autodoc]] MraForTokenClassification
    - forward

## MraForQuestionAnswering

[[autodoc]] MraForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "b8a75245-7fe3-4325-88c3-14fa0fd87bdb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "574e3ebf-3276-442e-895a-a443ca6c2147", "node_type": "1", "metadata": {}, "hash": "176a40ea098eb50a28e3baf6d0a48e55146e1f8f1b98c3bf3def86b5d3ff85c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Pop2Piano

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/spaces/sweetcocoa/pop2piano">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The Pop2Piano model was proposed in [Pop2Piano : Pop Audio-based Piano Cover Generation](https://arxiv.org/abs/2211.00895) by Jongho Choi and Kyogu Lee.

Piano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great 
expertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you 
can directly generate a cover from a song's audio waveform. It is the first model to directly generate a piano cover 
from pop audio without melody and chord extraction modules. 

Pop2Piano is an encoder-decoder Transformer model based on [T5](https://arxiv.org/pdf/1910.10683.pdf). The input audio 
is transformed to its waveform and passed to the encoder, which transforms it to a latent representation. The decoder 
uses these latent representations to generate token ids in an autoregressive way. Each token id corresponds to one of four 
different token types: time, velocity, note and 'special'. The token ids are then decoded to their equivalent MIDI file.

The abstract from the paper is the following:

*Piano covers of pop music are enjoyed by many people. However, the
task of automatically generating piano covers of pop music is still
understudied. This is partly due to the lack of synchronized
{Pop, Piano Cover} data pairs, which made it challenging to apply
the latest data-intensive deep learning-based methods. To leverage
the power of the data-driven approach, we make a large amount of
paired and synchronized {Pop, Piano Cover} data using an automated
pipeline. In this paper, we present Pop2Piano, a Transformer network
that generates piano covers given waveforms of pop music. To the best
of our knowledge, this is the first model to generate a piano cover
directly from pop audio without using melody and chord extraction
modules. We show that Pop2Piano, trained with our dataset, is capable
of producing plausible piano covers.*

This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/sweetcocoa/pop2piano).

## Usage tips

* To use Pop2Piano, you will need to install the 🤗 Transformers library, as well as the following third party modules:  
```bash
pip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy
```
Please note that you may need to restart your runtime after installation.
* Pop2Piano is an Encoder-Decoder based model like T5.
* Pop2Piano can be used to generate midi-audio files for a given audio sequence.
* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.
* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.


key: _node_content
string_value:
{"id_": "574e3ebf-3276-442e-895a-a443ca6c2147", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b8a75245-7fe3-4325-88c3-14fa0fd87bdb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "daaa5cae7e6631303a13c04b8cf402a3746450216cdc623e2bef542ea6371a70", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ce389f2-0600-43ad-a0c9-93bc67ad7701", "node_type": "1", "metadata": {}, "hash": "fecc9aabea6e4a742ba500b56d7e9862610312e364e7698c0b7d6adf18eca638", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1489, "end_char_idx": 4222, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is partly due to the lack of synchronized
{Pop, Piano Cover} data pairs, which made it challenging to apply
the latest data-intensive deep learning-based methods. To leverage
the power of the data-driven approach, we make a large amount of
paired and synchronized {Pop, Piano Cover} data using an automated
pipeline. In this paper, we present Pop2Piano, a Transformer network
that generates piano covers given waveforms of pop music. To the best
of our knowledge, this is the first model to generate a piano cover
directly from pop audio without using melody and chord extraction
modules. We show that Pop2Piano, trained with our dataset, is capable
of producing plausible piano covers.*

This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/sweetcocoa/pop2piano).

## Usage tips

* To use Pop2Piano, you will need to install the 🤗 Transformers library, as well as the following third party modules:  
```bash
pip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy
```
Please note that you may need to restart your runtime after installation.
* Pop2Piano is an Encoder-Decoder based model like T5.
* Pop2Piano can be used to generate midi-audio files for a given audio sequence.
* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.
* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.

## Examples

- Example using HuggingFace Dataset:

```python
>>> from datasets import load_dataset
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")
>>> ds = load_dataset("sweetcocoa/pop2piano_ci", split="test")

>>> inputs = processor(
...     audio=ds["audio"][0]["array"], sampling_rate=ds["audio"][0]["sampling_rate"], return_tensors="pt"
... )
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example using your own audio file:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> audio, sr = librosa.load("<your_audio_file_here>", sr=44100)  # feel free to change the sr to a suitable value.


key: _node_content
string_value:
{"id_": "3ce389f2-0600-43ad-a0c9-93bc67ad7701", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "574e3ebf-3276-442e-895a-a443ca6c2147", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1224b1715e5d3b4f0439e5c7635ea2316ded3ee9cb2d46011d22cc5bdf9cec5c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e190ac25-0c43-4572-8ac7-6538dd8e2f02", "node_type": "1", "metadata": {}, "hash": "98694f1f8b94235607e3f01d18ce2b306e6bd7554ca8122f8467b919d7bdd31b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2633, "end_char_idx": 4983, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* Pop2Piano is an Encoder-Decoder based model like T5.
* Pop2Piano can be used to generate midi-audio files for a given audio sequence.
* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.
* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.

## Examples

- Example using HuggingFace Dataset:

```python
>>> from datasets import load_dataset
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")
>>> ds = load_dataset("sweetcocoa/pop2piano_ci", split="test")

>>> inputs = processor(
...     audio=ds["audio"][0]["array"], sampling_rate=ds["audio"][0]["sampling_rate"], return_tensors="pt"
... )
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example using your own audio file:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> audio, sr = librosa.load("<your_audio_file_here>", sr=44100)  # feel free to change the sr to a suitable value.
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors="pt")
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example of processing multiple audio files in batch:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> # feel free to change the sr to a suitable value.


key: _node_content
string_value:
{"id_": "e190ac25-0c43-4572-8ac7-6538dd8e2f02", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ce389f2-0600-43ad-a0c9-93bc67ad7701", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "07f637235d8892095781781d037f44665c6de124549c5cf318f275052382ef8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ce118c8-b9da-41d0-8257-6f8d002e75de", "node_type": "1", "metadata": {}, "hash": "d111db0729e3d39edcd4585a8418407c6ab6d2c475215c82eb65fe86b5fc4bbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4223, "end_char_idx": 7426, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors="pt")
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example of processing multiple audio files in batch:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> # feel free to change the sr to a suitable value.
>>> audio1, sr1 = librosa.load("<your_first_audio_file_here>", sr=44100)  
>>> audio2, sr2 = librosa.load("<your_second_audio_file_here>", sr=44100)
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=[audio1, audio2], sampling_rate=[sr1, sr2], return_attention_mask=True, return_tensors="pt")
>>> # Since we now generating in batch(2 audios) we must pass the attention_mask
>>> model_output = model.generate(
...     input_features=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     composer="composer1",
... )
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"]

>>> # Since we now have 2 generated MIDI files
>>> tokenizer_output[0].write("./Outputs/midi_output1.mid")
>>> tokenizer_output[1].write("./Outputs/midi_output2.mid")
```


- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer

>>> # feel free to change the sr to a suitable value.
>>> audio1, sr1 = librosa.load("<your_first_audio_file_here>", sr=44100)  
>>> audio2, sr2 = librosa.load("<your_second_audio_file_here>", sr=44100)
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained("sweetcocoa/pop2piano")
>>> tokenizer = Pop2PianoTokenizer.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = feature_extractor(
...     audio=[audio1, audio2], 
...     sampling_rate=[sr1, sr2], 
...     return_attention_mask=True, 
...     return_tensors="pt",
... )
>>> # Since we now generating in batch(2 audios) we must pass the attention_mask
>>> model_output = model.generate(
...     input_features=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     composer="composer1",
... )
>>> tokenizer_output = tokenizer.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"]

>>> # Since we now have 2 generated MIDI files
>>> tokenizer_output[0].write("./Outputs/midi_output1.mid")
>>> tokenizer_output[1].write("./Outputs/midi_output2.mid")
```


key: _node_content
string_value:
{"id_": "1ce118c8-b9da-41d0-8257-6f8d002e75de", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fb5af5f-dd72-4c89-be67-84af2736028e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8415df4061d3a9de5365d85b38485cc2f07ce1c6ed9e2186eccae3b2f37f6834", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e190ac25-0c43-4572-8ac7-6538dd8e2f02", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano/", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0846bcdfcc20bea26d92e177381a011790c5e80f3eb660a11b14ec8796859ef6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5981, "end_char_idx": 7814, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer

>>> # feel free to change the sr to a suitable value.
>>> audio1, sr1 = librosa.load("<your_first_audio_file_here>", sr=44100)  
>>> audio2, sr2 = librosa.load("<your_second_audio_file_here>", sr=44100)
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained("sweetcocoa/pop2piano")
>>> tokenizer = Pop2PianoTokenizer.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = feature_extractor(
...     audio=[audio1, audio2], 
...     sampling_rate=[sr1, sr2], 
...     return_attention_mask=True, 
...     return_tensors="pt",
... )
>>> # Since we now generating in batch(2 audios) we must pass the attention_mask
>>> model_output = model.generate(
...     input_features=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     composer="composer1",
... )
>>> tokenizer_output = tokenizer.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"]

>>> # Since we now have 2 generated MIDI files
>>> tokenizer_output[0].write("./Outputs/midi_output1.mid")
>>> tokenizer_output[1].write("./Outputs/midi_output2.mid")
```


## Pop2PianoConfig

[[autodoc]] Pop2PianoConfig

## Pop2PianoFeatureExtractor

[[autodoc]] Pop2PianoFeatureExtractor
    - __call__

## Pop2PianoForConditionalGeneration

[[autodoc]] Pop2PianoForConditionalGeneration
    - forward
    - generate

## Pop2PianoTokenizer

[[autodoc]] Pop2PianoTokenizer
    - __call__

## Pop2PianoProcessor

[[autodoc]] Pop2PianoProcessor
    - __call__


key: _node_content
string_value:
{"id_": "e525289e-f5ba-44b8-9e90-a98f5e9de3ed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "38f108b559134f09e24ae7931982b4581c92563f9e6e9e8314c686099b900a9b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "047e1723-ca0f-4802-90ae-4967d806accf", "node_type": "1", "metadata": {}, "hash": "079ce304f51d159f26bd7a0e31e7fde21db4509f17d12205709b7e679b34f79f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ConvNeXt V2

## Overview

The ConvNeXt V2 model was proposed in [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://arxiv.org/abs/2301.00808) by Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, Saining Xie.
ConvNeXt V2 is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, and a successor of [ConvNeXT](convnext).

The abstract from the paper is the following:

*Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked  autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnextv2_architecture.png"
alt="drawing" width="600"/>

<small> ConvNeXt V2 architecture. Taken from the <a href="https://arxiv.org/abs/2301.00808">original paper</a>.</small>

This model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt-V2).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ConvNeXt V2.

<PipelineTag pipeline="image-classification"/>

- [`ConvNextV2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ConvNextV2Config

[[autodoc]] ConvNextV2Config

## ConvNextV2Model

[[autodoc]] ConvNextV2Model
    - forward

## ConvNextV2ForImageClassification

[[autodoc]] ConvNextV2ForImageClassification
    - forward

## TFConvNextV2Model

[[autodoc]] TFConvNextV2Model
    - call


key: _node_content
string_value:
{"id_": "047e1723-ca0f-4802-90ae-4967d806accf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a20dd9a-d460-44d9-95ae-89025c7a70d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "38f108b559134f09e24ae7931982b4581c92563f9e6e9e8314c686099b900a9b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e525289e-f5ba-44b8-9e90-a98f5e9de3ed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnextv2/", "title": "ConvNeXt V2", "tokens": 966, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "22e9f25bffb1f2ce07243f172206a1b189e6f46bde1ecc79b8e9e4f17fb47cb1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3249, "end_char_idx": 3345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "c2b03a1c-20b6-4848-873c-6935867617ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc855d8a-d686-47e7-8c5f-dd817429d83d", "node_type": "1", "metadata": {}, "hash": "5150e208065b58a2c9c3756a9d5c79d3cca0f3d2dee1f7f52a31e78e77a112d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3237, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Donut

## Overview

The Donut model was proposed in [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) by
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.
Donut consists of an image Transformer encoder and an autoregressive text Transformer decoder to perform document understanding
tasks such as document image classification, form understanding and visual question answering.

The abstract from the paper is the following:

*Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/donut_architecture.jpg"
alt="drawing" width="600"/>

<small> Donut high-level overview. Taken from the <a href="https://arxiv.org/abs/2111.15664">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found
[here](https://github.com/clovaai/donut).

## Usage tips

- The quickest way to get started with Donut is by checking the [tutorial
  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut), which show how to use the model
  at inference time as well as fine-tuning on custom data.
- Donut is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.

## Inference examples

Donut's [`VisionEncoderDecoder`] model accepts images as input and makes use of
[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.

The [`DonutImageProcessor`] class is responsible for preprocessing the input image and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`] decodes the generated target tokens to the target string. The
[`DonutProcessor`] wraps [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]
into a single instance to both extract the input features and decode the predicted token ids.


key: _node_content
string_value:
{"id_": "dc855d8a-d686-47e7-8c5f-dd817429d83d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2b03a1c-20b6-4848-873c-6935867617ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "aa308d903d3b68159d1953e904373fe7df4d02bba508341f0be3f244dfdd3b28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56b1a2f7-629e-4c38-987c-e09a34561221", "node_type": "1", "metadata": {}, "hash": "ebaa59bff908fade149153d343a38d8556911e98881af4ee450717dd2e82be83", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1715, "end_char_idx": 4745, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/donut_architecture.jpg"
alt="drawing" width="600"/>

<small> Donut high-level overview. Taken from the <a href="https://arxiv.org/abs/2111.15664">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found
[here](https://github.com/clovaai/donut).

## Usage tips

- The quickest way to get started with Donut is by checking the [tutorial
  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut), which show how to use the model
  at inference time as well as fine-tuning on custom data.
- Donut is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.

## Inference examples

Donut's [`VisionEncoderDecoder`] model accepts images as input and makes use of
[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.

The [`DonutImageProcessor`] class is responsible for preprocessing the input image and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`] decodes the generated target tokens to the target string. The
[`DonutProcessor`] wraps [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]
into a single instance to both extract the input features and decode the predicted token ids.

- Step-by-step Document Image Classification

```py
>>> import re

>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-rvlcdip")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-rvlcdip")

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)  # doctest: +IGNORE_RESULT

>>> # load document image
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[1]["image"]

>>> # prepare decoder inputs
>>> task_prompt = "<s_rvlcdip>"
>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids

>>> pixel_values = processor(image, return_tensors="pt").pixel_values

>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )

>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.


key: _node_content
string_value:
{"id_": "56b1a2f7-629e-4c38-987c-e09a34561221", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc855d8a-d686-47e7-8c5f-dd817429d83d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "28fd5347b67de6d67848bf69c3e6f03caed5e2554a447970e924f0b0f9ee3bbd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "21d11092-a4ae-47f4-826e-25e30d29fa9a", "node_type": "1", "metadata": {}, "hash": "0ae56e88d13670c9a9dcff7cad024723ecc5c51f867ae4b1dbfad946f98c5b9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3239, "end_char_idx": 6382, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Step-by-step Document Image Classification

```py
>>> import re

>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-rvlcdip")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-rvlcdip")

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)  # doctest: +IGNORE_RESULT

>>> # load document image
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[1]["image"]

>>> # prepare decoder inputs
>>> task_prompt = "<s_rvlcdip>"
>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids

>>> pixel_values = processor(image, return_tensors="pt").pixel_values

>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )

>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'class': 'advertisement'}
```

- Step-by-step Document Parsing

```py
>>> import re

>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)  # doctest: +IGNORE_RESULT

>>> # load document image
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[2]["image"]

>>> # prepare decoder inputs
>>> task_prompt = "<s_cord-v2>"
>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids

>>> pixel_values = processor(image, return_tensors="pt").pixel_values

>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )

>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.


key: _node_content
string_value:
{"id_": "21d11092-a4ae-47f4-826e-25e30d29fa9a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56b1a2f7-629e-4c38-987c-e09a34561221", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "53e3926a763c424fdd6db06a4b62f36b8df59a3058959587b56cec7f8e47a1f8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b2de6d9e-e8f3-4061-8cc0-92c795bcc5ed", "node_type": "1", "metadata": {}, "hash": "44d407268a4750aa1975cf042079097ed14e1cf00ef0b8c79213157a6c4e935e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4745, "end_char_idx": 7541, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'class': 'advertisement'}
```

- Step-by-step Document Parsing

```py
>>> import re

>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)  # doctest: +IGNORE_RESULT

>>> # load document image
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[2]["image"]

>>> # prepare decoder inputs
>>> task_prompt = "<s_cord-v2>"
>>> decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids

>>> pixel_values = processor(image, return_tensors="pt").pixel_values

>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )

>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000', 'cashprice': '20,000', 'changeprice': '3,000'}}
```

- Step-by-step Document Visual Question Answering (DocVQA)

```py
>>> import re

>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)  # doctest: +IGNORE_RESULT

>>> # load document image from the DocVQA dataset
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[0]["image"]

>>> # prepare decoder inputs
>>> task_prompt = "<s_docvqa><s_question>{user_input}</s_question><s_answer>"
>>> question = "When is the coffee break?"


key: _node_content
string_value:
{"id_": "b2de6d9e-e8f3-4061-8cc0-92c795bcc5ed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25eb9725-d9e5-42b5-8fbd-ef144ebbc5e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4016c563a7440df4116439594f374e2347d3532f646ea55f8b217347f7674966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "21d11092-a4ae-47f4-826e-25e30d29fa9a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/donut/", "title": "Donut", "tokens": 2384, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7269ccbd517bd838793b4dd149d17dec7d951e21c75602bdecb928aae9aa717f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6382, "end_char_idx": 9209, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'menu': {'nm': 'CINNAMON SUGAR', 'unitprice': '17,000', 'cnt': '1 x', 'price': '17,000'}, 'sub_total': {'subtotal_price': '17,000'}, 'total': {'total_price': '17,000', 'cashprice': '20,000', 'changeprice': '3,000'}}
```

- Step-by-step Document Visual Question Answering (DocVQA)

```py
>>> import re

>>> from transformers import DonutProcessor, VisionEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
>>> model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> model.to(device)  # doctest: +IGNORE_RESULT

>>> # load document image from the DocVQA dataset
>>> dataset = load_dataset("hf-internal-testing/example-documents", split="test")
>>> image = dataset[0]["image"]

>>> # prepare decoder inputs
>>> task_prompt = "<s_docvqa><s_question>{user_input}</s_question><s_answer>"
>>> question = "When is the coffee break?"
>>> prompt = task_prompt.replace("{user_input}", question)
>>> decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids

>>> pixel_values = processor(image, return_tensors="pt").pixel_values

>>> outputs = model.generate(
...     pixel_values.to(device),
...     decoder_input_ids=decoder_input_ids.to(device),
...     max_length=model.decoder.config.max_position_embeddings,
...     pad_token_id=processor.tokenizer.pad_token_id,
...     eos_token_id=processor.tokenizer.eos_token_id,
...     use_cache=True,
...     bad_words_ids=[[processor.tokenizer.unk_token_id]],
...     return_dict_in_generate=True,
... )

>>> sequence = processor.batch_decode(outputs.sequences)[0]
>>> sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
>>> sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
>>> print(processor.token2json(sequence))
{'question': 'When is the coffee break?', 'answer': '11-14 to 11:39 a.m.'}
```

See the [model hub](https://huggingface.co/models?filter=donut) to look for Donut checkpoints.

## Training

We refer to the [tutorial notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Donut).

## DonutSwinConfig

[[autodoc]] DonutSwinConfig

## DonutImageProcessor

[[autodoc]] DonutImageProcessor
    - preprocess

## DonutFeatureExtractor

[[autodoc]] DonutFeatureExtractor
    - __call__

## DonutProcessor

[[autodoc]] DonutProcessor
    - __call__
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## DonutSwinModel

[[autodoc]] DonutSwinModel
    - forward


key: _node_content
string_value:
{"id_": "5804001f-de9b-4d4e-9b33-09b34fb43351", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mluke/", "title": "mLUKE", "tokens": 722, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e2d1b7db-f527-4928-b7e7-6c0e36afcc3b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mluke/", "title": "mLUKE", "tokens": 722, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "626d8ad3a7f71f87f96d7ff03817cabc3a903c786e6a001d2c926f31ca207f05", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2471, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# mLUKE

## Overview

The mLUKE model was proposed in [mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](https://arxiv.org/abs/2110.08151) by Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. It's a multilingual extension
of the [LUKE model](https://arxiv.org/abs/2010.01057) trained on the basis of XLM-RoBERTa.

It is based on XLM-RoBERTa and adds entity embeddings, which helps improve performance on various downstream tasks
involving reasoning about entities such as named entity recognition, extractive question answering, relation
classification, cloze-style knowledge completion.

The abstract from the paper is the following:

*Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual
alignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining
and do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging
entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages
with entity representations and show the model consistently outperforms word-based pretrained models in various
cross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity
representations into the input allows us to extract more language-agnostic features. We also evaluate the model with a
multilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits correct factual
knowledge more likely than using only word representations.*

This model was contributed by [ryo0634](https://huggingface.co/ryo0634). The original code can be found [here](https://github.com/studio-ousia/luke).

## Usage tips

One can directly plug in the weights of mLUKE into a LUKE model, like so:

```python
from transformers import LukeModel

model = LukeModel.from_pretrained("studio-ousia/mluke-base")
```

Note that mLUKE has its own tokenizer, [`MLukeTokenizer`]. You can initialize it as follows:

```python
from transformers import MLukeTokenizer

tokenizer = MLukeTokenizer.from_pretrained("studio-ousia/mluke-base")
```

<Tip>

As mLUKE's architecture is equivalent to that of LUKE, one can refer to [LUKE's documentation page](luke) for all
tips, code examples and notebooks.

</Tip>

## MLukeTokenizer

[[autodoc]] MLukeTokenizer
    - __call__
    - save_vocabulary


key: _node_content
string_value:
{"id_": "df73df6f-7e52-465e-80c7-914a99ee1535", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ae56397-8753-45f1-abd5-21401d75d964", "node_type": "1", "metadata": {}, "hash": "d02c7793a966c9e1c22ced7965c51a1d9845be5fcfb0e57cf219e9f028a4fd01", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3038, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# QDQBERT

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The QDQBERT model can be referenced in [Integer Quantization for Deep Learning Inference: Principles and Empirical
Evaluation](https://arxiv.org/abs/2004.09602) by Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev and Paulius
Micikevicius.

The abstract from the paper is the following:

*Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by
taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of
quantization parameters and evaluate their choices on a wide range of neural network models for different application
domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration
by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is
able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are
more difficult to quantize, such as MobileNets and BERT-large.*

This model was contributed by [shangz](https://huggingface.co/shangz).

## Usage tips

- QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to (i) linear layer
  inputs and weights, (ii) matmul inputs, (iii) residual add inputs, in BERT model.
- QDQBERT requires the dependency of [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). To install `pip install pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com`
- QDQBERT model can be loaded from any checkpoint of HuggingFace BERT model (for example *google-bert/bert-base-uncased*), and
  perform Quantization Aware Training/Post Training Quantization.
- A complete example of using QDQBERT model to perform Quatization Aware Training and Post Training Quantization for
  SQUAD task can be found at [transformers/examples/research_projects/quantization-qdqbert/](examples/research_projects/quantization-qdqbert/).

### Set default quantizers

QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to BERT by
`TensorQuantizer` in [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). `TensorQuantizer` is the module
for quantizing tensors, with `QuantDescriptor` defining how the tensor should be quantized. Refer to [Pytorch
Quantization Toolkit userguide](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html) for more details.

Before creating QDQBERT model, one has to set the default `QuantDescriptor` defining default tensor quantizers.


key: _node_content
string_value:
{"id_": "3ae56397-8753-45f1-abd5-21401d75d964", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df73df6f-7e52-465e-80c7-914a99ee1535", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5002698f76f6f278b8315fbb01083029a901a5cd0aa89e2fc428c4f7b96b631d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c758f6fe-dded-4129-aa6d-5a6d1602673e", "node_type": "1", "metadata": {}, "hash": "b0f3520e3bc68e6f7e18b8fb72f58f64bebf58b5a7697edc39fa8cdffd565b90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1471, "end_char_idx": 4746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips

- QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to (i) linear layer
  inputs and weights, (ii) matmul inputs, (iii) residual add inputs, in BERT model.
- QDQBERT requires the dependency of [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). To install `pip install pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com`
- QDQBERT model can be loaded from any checkpoint of HuggingFace BERT model (for example *google-bert/bert-base-uncased*), and
  perform Quantization Aware Training/Post Training Quantization.
- A complete example of using QDQBERT model to perform Quatization Aware Training and Post Training Quantization for
  SQUAD task can be found at [transformers/examples/research_projects/quantization-qdqbert/](examples/research_projects/quantization-qdqbert/).

### Set default quantizers

QDQBERT model adds fake quantization operations (pair of QuantizeLinear/DequantizeLinear ops) to BERT by
`TensorQuantizer` in [Pytorch Quantization Toolkit](https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization). `TensorQuantizer` is the module
for quantizing tensors, with `QuantDescriptor` defining how the tensor should be quantized. Refer to [Pytorch
Quantization Toolkit userguide](https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html) for more details.

Before creating QDQBERT model, one has to set the default `QuantDescriptor` defining default tensor quantizers.

Example:

```python
>>> import pytorch_quantization.nn as quant_nn
>>> from pytorch_quantization.tensor_quant import QuantDescriptor

>>> # The default tensor quantizer is set to use Max calibration method
>>> input_desc = QuantDescriptor(num_bits=8, calib_method="max")
>>> # The default tensor quantizer is set to be per-channel quantization for weights
>>> weight_desc = QuantDescriptor(num_bits=8, axis=((0,)))
>>> quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)
>>> quant_nn.QuantLinear.set_default_quant_desc_weight(weight_desc)
```

### Calibration

Calibration is the terminology of passing data samples to the quantizer and deciding the best scaling factors for
tensors. After setting up the tensor quantizers, one can use the following example to calibrate the model:

```python
>>> # Find the TensorQuantizer and enable calibration
>>> for name, module in model.named_modules():
...     if name.endswith("_input_quantizer"):
...         module.enable_calib()
...         module.disable_quant()  # Use full precision data to calibrate

>>> # Feeding data samples
>>> model(x)
>>> # ...

>>> # Finalize calibration
>>> for name, module in model.named_modules():
...     if name.endswith("_input_quantizer"):
...         module.load_calib_amax()
...         module.enable_quant()

>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process
>>> model.cuda()

>>> # Keep running the quantized model
>>> # ...
```

### Export to ONNX

The goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake
quantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops.


key: _node_content
string_value:
{"id_": "c758f6fe-dded-4129-aa6d-5a6d1602673e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ae56397-8753-45f1-abd5-21401d75d964", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d444518b5e7c58447c871653222cc1c56aec21cdc10ca00e760559cd14e6ca58", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1793304a-2547-4b61-908c-d077da41065d", "node_type": "1", "metadata": {}, "hash": "94d669c0582a3a62ad85cf27e03dbb584c20ba09374ff128c865a1a9938ac725", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3040, "end_char_idx": 4973, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Example:

```python
>>> import pytorch_quantization.nn as quant_nn
>>> from pytorch_quantization.tensor_quant import QuantDescriptor

>>> # The default tensor quantizer is set to use Max calibration method
>>> input_desc = QuantDescriptor(num_bits=8, calib_method="max")
>>> # The default tensor quantizer is set to be per-channel quantization for weights
>>> weight_desc = QuantDescriptor(num_bits=8, axis=((0,)))
>>> quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)
>>> quant_nn.QuantLinear.set_default_quant_desc_weight(weight_desc)
```

### Calibration

Calibration is the terminology of passing data samples to the quantizer and deciding the best scaling factors for
tensors. After setting up the tensor quantizers, one can use the following example to calibrate the model:

```python
>>> # Find the TensorQuantizer and enable calibration
>>> for name, module in model.named_modules():
...     if name.endswith("_input_quantizer"):
...         module.enable_calib()
...         module.disable_quant()  # Use full precision data to calibrate

>>> # Feeding data samples
>>> model(x)
>>> # ...

>>> # Finalize calibration
>>> for name, module in model.named_modules():
...     if name.endswith("_input_quantizer"):
...         module.load_calib_amax()
...         module.enable_quant()

>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process
>>> model.cuda()

>>> # Keep running the quantized model
>>> # ...
```

### Export to ONNX

The goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake
quantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops. After setting static member of
TensorQuantizer to use Pytorch’s own fake quantization functions, fake quantized model can be exported to ONNX, follow
the instructions in [torch.onnx](https://pytorch.org/docs/stable/onnx.html).


key: _node_content
string_value:
{"id_": "1793304a-2547-4b61-908c-d077da41065d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c96e7ff9-5fdb-4083-a8ff-bb8f60387115", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0d095170027a7cdff49f30177ff6088b23f5c9d2fb1716f4393582fe8da6d6d6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c758f6fe-dded-4129-aa6d-5a6d1602673e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qdqbert/", "title": "QDQBERT", "tokens": 1678, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2e10e9955d73070cfbd45b8dd439bca0d531226326f34b81f2ebd495b462fc05", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3734, "end_char_idx": 6292, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
After setting up the tensor quantizers, one can use the following example to calibrate the model:

```python
>>> # Find the TensorQuantizer and enable calibration
>>> for name, module in model.named_modules():
...     if name.endswith("_input_quantizer"):
...         module.enable_calib()
...         module.disable_quant()  # Use full precision data to calibrate

>>> # Feeding data samples
>>> model(x)
>>> # ...

>>> # Finalize calibration
>>> for name, module in model.named_modules():
...     if name.endswith("_input_quantizer"):
...         module.load_calib_amax()
...         module.enable_quant()

>>> # If running on GPU, it needs to call .cuda() again because new tensors will be created by calibration process
>>> model.cuda()

>>> # Keep running the quantized model
>>> # ...
```

### Export to ONNX

The goal of exporting to ONNX is to deploy inference by [TensorRT](https://developer.nvidia.com/tensorrt). Fake
quantization will be broken into a pair of QuantizeLinear/DequantizeLinear ONNX ops. After setting static member of
TensorQuantizer to use Pytorch’s own fake quantization functions, fake quantized model can be exported to ONNX, follow
the instructions in [torch.onnx](https://pytorch.org/docs/stable/onnx.html). Example:

```python
>>> from pytorch_quantization.nn import TensorQuantizer

>>> TensorQuantizer.use_fb_fake_quant = True

>>> # Load the calibrated model
>>> ...
>>> # ONNX export
>>> torch.onnx.export(...)
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## QDQBertConfig

[[autodoc]] QDQBertConfig

## QDQBertModel

[[autodoc]] QDQBertModel
    - forward

## QDQBertLMHeadModel

[[autodoc]] QDQBertLMHeadModel
    - forward

## QDQBertForMaskedLM

[[autodoc]] QDQBertForMaskedLM
    - forward

## QDQBertForSequenceClassification

[[autodoc]] QDQBertForSequenceClassification
    - forward

## QDQBertForNextSentencePrediction

[[autodoc]] QDQBertForNextSentencePrediction
    - forward

## QDQBertForMultipleChoice

[[autodoc]] QDQBertForMultipleChoice
    - forward

## QDQBertForTokenClassification

[[autodoc]] QDQBertForTokenClassification
    - forward

## QDQBertForQuestionAnswering

[[autodoc]] QDQBertForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "5a0e3a96-a426-4be3-8f4e-ada8d277259f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d98f8565-408c-4a39-8983-6e4022852f85", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1c86cf30c271c0ce0c51e89f86532dedc837b48365ec8370423902f560a9e5e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00cb74b5-33d1-4314-95f2-e5507afd8ef7", "node_type": "1", "metadata": {}, "hash": "c58b9b3cc7d9f7c5e20fa00d0f57f2c6cb7d0fcfe6c7023bdd60d970576aaf52", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BertGeneration

## Overview

The BertGeneration model is a BERT model that can be leveraged for sequence-to-sequence tasks using
[`EncoderDecoderModel`] as proposed in [Leveraging Pre-trained Checkpoints for Sequence Generation
Tasks](https://arxiv.org/abs/1907.12461) by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.

The abstract from the paper is the following:

*Unsupervised pretraining of large neural models has recently revolutionized Natural Language Processing. By
warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple
benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language
Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We
developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT,
GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both
encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation,
Text Summarization, Sentence Splitting, and Sentence Fusion.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder).

## Usage examples and tips

The model can be used in combination with the [`EncoderDecoderModel`] to leverage two pretrained BERT checkpoints for 
subsequent fine-tuning:

```python
>>> # leverage checkpoints for Bert2Bert model...
>>> # use BERT's cls token as BOS token and sep token as EOS token
>>> encoder = BertGenerationEncoder.from_pretrained("google-bert/bert-large-uncased", bos_token_id=101, eos_token_id=102)
>>> # add cross attention layers and use BERT's cls token as BOS token and sep token as EOS token
>>> decoder = BertGenerationDecoder.from_pretrained(
...     "google-bert/bert-large-uncased", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102
... )
>>> bert2bert = EncoderDecoderModel(encoder=encoder, decoder=decoder)

>>> # create tokenizer...
>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-large-uncased")

>>> input_ids = tokenizer(
...     "This is a long article to summarize", add_special_tokens=False, return_tensors="pt"
... ).input_ids
>>> labels = tokenizer("This is a short summary", return_tensors="pt").input_ids

>>> # train...
>>> loss = bert2bert(input_ids=input_ids, decoder_input_ids=labels, labels=labels).loss
>>> loss.backward()
```

Pretrained [`EncoderDecoderModel`] are also directly available in the model hub, e.g.:

```python
>>> # instantiate sentence fusion model
>>> sentence_fuser = EncoderDecoderModel.from_pretrained("google/roberta2roberta_L-24_discofuse")
>>> tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_discofuse")

>>> input_ids = tokenizer(
...     "This is the first sentence. This is the second sentence.", add_special_tokens=False, return_tensors="pt"
... ).input_ids

>>> outputs = sentence_fuser.generate(input_ids)

>>> print(tokenizer.decode(outputs[0]))
```

Tips:

- [`BertGenerationEncoder`] and [`BertGenerationDecoder`] should be used in
  combination with [`EncoderDecoder`].


key: _node_content
string_value:
{"id_": "00cb74b5-33d1-4314-95f2-e5507afd8ef7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d98f8565-408c-4a39-8983-6e4022852f85", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1c86cf30c271c0ce0c51e89f86532dedc837b48365ec8370423902f560a9e5e2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a0e3a96-a426-4be3-8f4e-ada8d277259f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert-generation/", "title": "BertGeneration", "tokens": 1045, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2f9e72c01730cecf737d844d32f53015442bcf6cc224c8862fc7873538d78270", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2767, "end_char_idx": 3863, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
:

```python
>>> # instantiate sentence fusion model
>>> sentence_fuser = EncoderDecoderModel.from_pretrained("google/roberta2roberta_L-24_discofuse")
>>> tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_discofuse")

>>> input_ids = tokenizer(
...     "This is the first sentence. This is the second sentence.", add_special_tokens=False, return_tensors="pt"
... ).input_ids

>>> outputs = sentence_fuser.generate(input_ids)

>>> print(tokenizer.decode(outputs[0]))
```

Tips:

- [`BertGenerationEncoder`] and [`BertGenerationDecoder`] should be used in
  combination with [`EncoderDecoder`].
- For summarization, sentence splitting, sentence fusion and translation, no special tokens are required for the input.
  Therefore, no EOS token should be added to the end of the input.

## BertGenerationConfig

[[autodoc]] BertGenerationConfig

## BertGenerationTokenizer

[[autodoc]] BertGenerationTokenizer
    - save_vocabulary

## BertGenerationEncoder

[[autodoc]] BertGenerationEncoder
    - forward

## BertGenerationDecoder

[[autodoc]] BertGenerationDecoder
    - forward


key: _node_content
string_value:
{"id_": "b035a8be-30c7-46cc-972e-a342e23c7d39", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a5c23b428571e98b67a1340548c16bc1e47739fe122bf0a4eaedca03e3a8146e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8fbb773-0f03-4b16-aea0-2cf4a7901c8a", "node_type": "1", "metadata": {}, "hash": "a5607f6ecfb741e936af0da4910c5c2d5241e33ab33b18306371b5f0c6001b85", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2921, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Transformer XL

<Tip warning={true}>

This model is in maintenance mode only, so we won't accept any new PRs changing its code. This model was deprecated due to security issues linked to `pickle.load`.

We recommend switching to more recent models for improved security.

In case you would still like to use `TransfoXL` in your experiments, we recommend using the [Hub checkpoint](https://huggingface.co/transfo-xl/transfo-xl-wt103) with a specific revision to ensure you are downloading safe files from the Hub.

You will need to set the environment variable `TRUST_REMOTE_CODE` to `True` in order to allow the
usage of `pickle.load()`:

```python
import os
from transformers import TransfoXLTokenizer, TransfoXLLMHeadModel

os.environ["TRUST_REMOTE_CODE"] = "True"

checkpoint = 'transfo-xl/transfo-xl-wt103'
revision = '40a186da79458c9f9de846edfaea79c412137f97'

tokenizer = TransfoXLTokenizer.from_pretrained(checkpoint, revision=revision)
model = TransfoXLLMHeadModel.from_pretrained(checkpoint, revision=revision)
```

If you run into any issues running this model, please reinstall the last version that supported this model: v4.35.0.
You can do so by running the following command: `pip install -U transformers==4.35.0`.

</Tip>

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=transfo-xl">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-transfo--xl-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/transfo-xl-wt103">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The Transformer-XL model was proposed in [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan
Salakhutdinov. It's a causal (uni-directional) transformer with relative positioning (sinusoïdal) embeddings which can
reuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax
inputs and outputs (tied).

The abstract from the paper is the following:

*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the
setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency
beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a
novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the
context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%
longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+
times faster than vanilla Transformers during evaluation.


key: _node_content
string_value:
{"id_": "f8fbb773-0f03-4b16-aea0-2cf4a7901c8a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a5c23b428571e98b67a1340548c16bc1e47739fe122bf0a4eaedca03e3a8146e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b035a8be-30c7-46cc-972e-a342e23c7d39", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4c419e35089845185ce5dc91c944699eb9fb7bc6ab7d5056836a7ab38b3ad234", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "979b9866-ce56-4298-9e26-31c937319d7c", "node_type": "1", "metadata": {}, "hash": "1c7c7442136b16edd0ce74dbb3c2745722cc5cfe04297b008c536f202bf9950e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1868, "end_char_idx": 4701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It's a causal (uni-directional) transformer with relative positioning (sinusoïdal) embeddings which can
reuse previously computed hidden-states to attend to longer context (memory). This model also uses adaptive softmax
inputs and outputs (tied).

The abstract from the paper is the following:

*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the
setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency
beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a
novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the
context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%
longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+
times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of
bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn
Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably
coherent, novel text articles with thousands of tokens.*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/kimiyoung/transformer-xl).

## Usage tips

- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The
  original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.
- Transformer-XL is one of the few models that has no sequence length limit.
- Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that may span across multiple documents, and segments are fed in order to the model.
- Basically, the hidden states of the previous segment are concatenated to the current input to compute the attention scores. This allows the model to pay attention to information that was in the previous segment as well as the current one. By stacking multiple attention layers, the receptive field can be increased to multiple previous segments.
- This changes the positional embeddings to positional relative embeddings (as the regular positional embeddings would give the same results in the current input and the current hidden state at a given position) and needs to make some adjustments in the way attention scores are computed.


key: _node_content
string_value:
{"id_": "979b9866-ce56-4298-9e26-31c937319d7c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0cf4b87-3f26-4ce2-af82-7f74ab96e191", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a5c23b428571e98b67a1340548c16bc1e47739fe122bf0a4eaedca03e3a8146e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8fbb773-0f03-4b16-aea0-2cf4a7901c8a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/transfo-xl/", "title": "Transformer XL", "tokens": 1679, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4bf8e9664e7afdaa29edea3382a4f49f960e5ded173ca64b5c5e7c9de0e97e90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2922, "end_char_idx": 6171, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Notably, we improve the state-of-the-art results of
bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn
Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably
coherent, novel text articles with thousands of tokens.*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/kimiyoung/transformer-xl).

## Usage tips

- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The
  original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.
- Transformer-XL is one of the few models that has no sequence length limit.
- Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that may span across multiple documents, and segments are fed in order to the model.
- Basically, the hidden states of the previous segment are concatenated to the current input to compute the attention scores. This allows the model to pay attention to information that was in the previous segment as well as the current one. By stacking multiple attention layers, the receptive field can be increased to multiple previous segments.
- This changes the positional embeddings to positional relative embeddings (as the regular positional embeddings would give the same results in the current input and the current hidden state at a given position) and needs to make some adjustments in the way attention scores are computed.


<Tip warning={true}>

TransformerXL does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035)

</Tip>

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## TransfoXLConfig

[[autodoc]] TransfoXLConfig

## TransfoXLTokenizer

[[autodoc]] TransfoXLTokenizer
    - save_vocabulary

## TransfoXL specific outputs

[[autodoc]] models.deprecated.transfo_xl.modeling_transfo_xl.TransfoXLModelOutput

[[autodoc]] models.deprecated.transfo_xl.modeling_transfo_xl.TransfoXLLMHeadModelOutput

[[autodoc]] models.deprecated.transfo_xl.modeling_tf_transfo_xl.TFTransfoXLModelOutput

[[autodoc]] models.deprecated.transfo_xl.modeling_tf_transfo_xl.TFTransfoXLLMHeadModelOutput

<frameworkcontent>
<pt>

## TransfoXLModel

[[autodoc]] TransfoXLModel
    - forward

## TransfoXLLMHeadModel

[[autodoc]] TransfoXLLMHeadModel
    - forward

## TransfoXLForSequenceClassification

[[autodoc]] TransfoXLForSequenceClassification
    - forward

</pt>
<tf>

## TFTransfoXLModel

[[autodoc]] TFTransfoXLModel
    - call

## TFTransfoXLLMHeadModel

[[autodoc]] TFTransfoXLLMHeadModel
    - call

## TFTransfoXLForSequenceClassification

[[autodoc]] TFTransfoXLForSequenceClassification
    - call

</tf>
</frameworkcontent>

## Internal Layers

[[autodoc]] AdaptiveEmbedding

[[autodoc]] TFAdaptiveEmbedding


key: _node_content
string_value:
{"id_": "c2a5eaf6-e041-459d-9f7c-772d2b140a10", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8da76146238a91143323bfab3c2d96a774f6ce96fe423a32ccb7db6c38ab7f37", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "175c7631-c1c2-40f2-9cbe-effd1abe1b62", "node_type": "1", "metadata": {}, "hash": "942e832408de5b9f8a5d478a15063d8e2fa0c770fa1399e90031e39bcf7a1b34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3403, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DETA

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The DETA model was proposed in [NMS Strikes Back](https://arxiv.org/abs/2212.06137) by Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, Philipp Krähenbühl.
DETA (short for Detection Transformers with Assignment) improves [Deformable DETR](deformable_detr) by replacing the one-to-one bipartite Hungarian matching loss
with one-to-many label assignments used in traditional detectors with non-maximum suppression (NMS). This leads to significant gains of up to 2.5 mAP.

The abstract from the paper is the following:

*Detection Transformer (DETR) directly transforms queries to unique objects by using one-to-one bipartite matching during training and enables end-to-end object detection. Recently, these models have surpassed traditional detectors on COCO with undeniable elegance. However, they differ from traditional detectors in multiple designs, including model architecture and training schedules, and thus the effectiveness of one-to-one matching is not fully understood. In this work, we conduct a strict comparison between the one-to-one Hungarian matching in DETRs and the one-to-many label assignments in traditional detectors with non-maximum supervision (NMS). Surprisingly, we observe one-to-many assignments with NMS consistently outperform standard one-to-one matching under the same setting, with a significant gain of up to 2.5 mAP. Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deta_architecture.jpg"
alt="drawing" width="600"/>

<small> DETA overview. Taken from the <a href="https://arxiv.org/abs/2212.06137">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/jozhang97/DETA).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DETA.

- Demo notebooks for DETA can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETA).
- Scripts for finetuning [`DetaForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "175c7631-c1c2-40f2-9cbe-effd1abe1b62", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "abd5b4a8-c91a-440e-9fd0-3019fd2b5821", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8da76146238a91143323bfab3c2d96a774f6ce96fe423a32ccb7db6c38ab7f37", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2a5eaf6-e041-459d-9f7c-772d2b140a10", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deta/", "title": "DETA", "tokens": 1005, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "11261e1bae4eb721df448a522e38c4cfb417926f09da1e619b58090b37ac3d34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1696, "end_char_idx": 3678, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Our detector that trains Deformable-DETR with traditional IoU-based label assignment achieved 50.2 COCO mAP within 12 epochs (1x schedule) with ResNet50 backbone, outperforming all existing traditional or transformer-based detectors in this setting. On multiple datasets, schedules, and architectures, we consistently show bipartite matching is unnecessary for performant detection transformers. Furthermore, we attribute the success of detection transformers to their expressive transformer architecture.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/deta_architecture.jpg"
alt="drawing" width="600"/>

<small> DETA overview. Taken from the <a href="https://arxiv.org/abs/2212.06137">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/jozhang97/DETA).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DETA.

- Demo notebooks for DETA can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETA).
- Scripts for finetuning [`DetaForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DetaConfig

[[autodoc]] DetaConfig

## DetaImageProcessor

[[autodoc]] DetaImageProcessor
    - preprocess
    - post_process_object_detection

## DetaModel

[[autodoc]] DetaModel
    - forward

## DetaForObjectDetection

[[autodoc]] DetaForObjectDetection
    - forward


key: _node_content
string_value:
{"id_": "0586bc02-81d7-4607-9d16-6d3dfad06b99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0bb1b753da9eed875f21f2085c22caa72082d5648cabad7f6d548e82219c987c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "252e1bf1-b470-4a02-9747-145862fbe2f6", "node_type": "1", "metadata": {}, "hash": "a64dc0b39a2a57ffb34c1d76c9296ba482aa2d91a2cabfa9c201ff13861ee2c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2653, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Starcoder2

## Overview

StarCoder2 is a family of open LLMs for code and comes in 3 different sizes with 3B, 7B and 15B parameters. The flagship StarCoder2-15B model is trained on over 4 trillion tokens and 600+ programming languages from The Stack v2. All models use Grouped Query Attention, a context window of 16,384 tokens with a sliding window attention of 4,096 tokens, and were trained using the Fill-in-the-Middle objective. The models have been released with the paper [StarCoder 2 and The Stack v2: The Next Generation](https://arxiv.org/abs/2402.19173) by Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.

The abstract of the paper is the following:

> The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size.


key: _node_content
string_value:
{"id_": "252e1bf1-b470-4a02-9747-145862fbe2f6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "adde3766-db47-47cd-80b3-8f5a902ac4c1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0bb1b753da9eed875f21f2085c22caa72082d5648cabad7f6d548e82219c987c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0586bc02-81d7-4607-9d16-6d3dfad06b99", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/starcoder2/", "title": "Starcoder2", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "25be629165bf984a9789512a575b2e95dd86c06801c524f325ece94acd749ada", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1547, "end_char_idx": 4578, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract of the paper is the following:

> The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data.
## License

The models are licensed under the [BigCode OpenRAIL-M v1 license agreement](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).
 
## Usage tips

The StarCoder2 models can be found in the [HuggingFace hub](https://huggingface.co/collections/bigcode/starcoder2-65de6da6e87db3383572be1a). You can find some examples for inference and fine-tuning in StarCoder2's [GitHub repo](https://github.com/bigcode-project/starcoder2).

These ready-to-use checkpoints can be downloaded and used via the HuggingFace Hub:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("bigcode/starcoder2-7b", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("bigcode/starcoder2-7b")

>>> prompt = "def print_hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=10, do_sample=False)
>>> tokenizer.batch_decode(generated_ids)[0]
'def print_hello_world():\n    print("Hello World!")\n\ndef print'
```

## Starcoder2Config

[[autodoc]] Starcoder2Config

## Starcoder2Model

[[autodoc]] Starcoder2Model
    - forward

## Starcoder2ForCausalLM

[[autodoc]] Starcoder2ForCausalLM
    - forward

## Starcoder2ForSequenceClassification

[[autodoc]] Starcoder2ForSequenceClassification
    - forward

## Starcoder2ForTokenClassification

[[autodoc]] Starcoder2ForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "3d3578a7-b019-4be8-a97d-5002ada99a5d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0f45d539-275f-4872-9731-28fc33993506", "node_type": "1", "metadata": {}, "hash": "4ee65d2f4c8537c448c77e9c2366a62cb1168d338e2ed0bb0d815c8f26d30eb0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3307, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ELECTRA

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=electra">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-electra-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/electra_large_discriminator_squad2_512">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The ELECTRA model was proposed in the paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than
Generators](https://openreview.net/pdf?id=r1xMH1BtvB). ELECTRA is a new pretraining approach which trains two
transformer models: the generator and the discriminator. The generator's role is to replace tokens in a sequence, and
is therefore trained as a masked language model. The discriminator, which is the model we're interested in, tries to
identify which tokens were replaced by the generator in the sequence.

The abstract from the paper is the following:

*Masked language modeling (MLM) pretraining methods such as BERT corrupt the input by replacing some tokens with [MASK]
and then train a model to reconstruct the original tokens. While they produce good results when transferred to
downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a
more sample-efficient pretraining task called replaced token detection. Instead of masking the input, our approach
corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead
of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that
predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments
demonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens
rather than just the small subset that was masked out. As a result, the contextual representations learned by our
approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are
particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained
using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,
where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when
using the same amount of compute.*

This model was contributed by [lysandre](https://huggingface.co/lysandre). The original code can be found [here](https://github.com/google-research/electra).

## Usage tips

- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The
  only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,
  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their
  embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection
  layer is used.
- ELECTRA is a transformer model pretrained with the use of another (small) masked language model.


key: _node_content
string_value:
{"id_": "0f45d539-275f-4872-9731-28fc33993506", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3d3578a7-b019-4be8-a97d-5002ada99a5d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8e05a8d1c792542359200934293b905b154f41fa1529f30d880357b1b453be6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a034c1c-1e0b-4b3b-8f3f-d443013e8633", "node_type": "1", "metadata": {}, "hash": "d56f0afe7b1d566508da7dc2f7bd94833395dcd218b5aaeadc147c935a3e9060", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then, instead
of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that
predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments
demonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens
rather than just the small subset that was masked out. As a result, the contextual representations learned by our
approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are
particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained
using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,
where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when
using the same amount of compute.*

This model was contributed by [lysandre](https://huggingface.co/lysandre). The original code can be found [here](https://github.com/google-research/electra).

## Usage tips

- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The
  only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,
  while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their
  embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection
  layer is used.
- ELECTRA is a transformer model pretrained with the use of another (small) masked language model. The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict which token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.
- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)
  contain both the generator and discriminator. The conversion script requires the user to name which model to export
  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all
  available ELECTRA models, however. This means that the discriminator may be loaded in the
  [`ElectraForMaskedLM`] model, and the generator may be loaded in the
  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it
  doesn't exist in the generator).

## Resources

- [Text classification task guide](./tasks/sequence_classification)
- [Token classification task guide](./tasks/token_classification)
- [Question answering task guide](./tasks/question_answering)
- [Causal language modeling task guide](./tasks/language_modeling)
- [Masked language modeling task guide](./tasks/masked_language_modeling)
- [Multiple choice task guide](./tasks/multiple_choice)

## ElectraConfig

[[autodoc]] ElectraConfig

## ElectraTokenizer

[[autodoc]] ElectraTokenizer

## ElectraTokenizerFast

[[autodoc]] ElectraTokenizerFast

## Electra specific outputs

[[autodoc]] models.electra.modeling_electra.


key: _node_content
string_value:
{"id_": "6a034c1c-1e0b-4b3b-8f3f-d443013e8633", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f45d539-275f-4872-9731-28fc33993506", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d2f6e3a303c26607ab87798c2b358820a680fd035a7a75c03762b4994cac233c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d534999f-ee3e-4a1b-b132-14ebd3b6d39f", "node_type": "1", "metadata": {}, "hash": "cf452d2274695614f192213ebfad351cbdf3a420e255c6d895d3e0e165190b1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The inputs are corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA has to predict which token is an original and which one has been replaced. Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.
- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)
  contain both the generator and discriminator. The conversion script requires the user to name which model to export
  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all
  available ELECTRA models, however. This means that the discriminator may be loaded in the
  [`ElectraForMaskedLM`] model, and the generator may be loaded in the
  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it
  doesn't exist in the generator).

## Resources

- [Text classification task guide](./tasks/sequence_classification)
- [Token classification task guide](./tasks/token_classification)
- [Question answering task guide](./tasks/question_answering)
- [Causal language modeling task guide](./tasks/language_modeling)
- [Masked language modeling task guide](./tasks/masked_language_modeling)
- [Multiple choice task guide](./tasks/multiple_choice)

## ElectraConfig

[[autodoc]] ElectraConfig

## ElectraTokenizer

[[autodoc]] ElectraTokenizer

## ElectraTokenizerFast

[[autodoc]] ElectraTokenizerFast

## Electra specific outputs

[[autodoc]] models.electra.modeling_electra.ElectraForPreTrainingOutput

[[autodoc]] models.electra.modeling_tf_electra.


key: _node_content
string_value:
{"id_": "d534999f-ee3e-4a1b-b132-14ebd3b6d39f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9b066302-c0a9-4902-a10b-fe696994d7fc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a3e994d5e87979f2434c894341a3ac221a205229781257cc862731de6150126", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a034c1c-1e0b-4b3b-8f3f-d443013e8633", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/electra/", "title": "ELECTRA", "tokens": 1919, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "38766b8def14cb05264103e83f243c909706c33a7a6e29b4a307af9e9886515d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Like for GAN training, the small language model is trained for a few steps (but with the original texts as objective, not to fool the ELECTRA model like in a traditional GAN setting) then the ELECTRA model is trained for a few steps.
- The ELECTRA checkpoints saved using [Google Research's implementation](https://github.com/google-research/electra)
  contain both the generator and discriminator. The conversion script requires the user to name which model to export
  into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all
  available ELECTRA models, however. This means that the discriminator may be loaded in the
  [`ElectraForMaskedLM`] model, and the generator may be loaded in the
  [`ElectraForPreTraining`] model (the classification head will be randomly initialized as it
  doesn't exist in the generator).

## Resources

- [Text classification task guide](./tasks/sequence_classification)
- [Token classification task guide](./tasks/token_classification)
- [Question answering task guide](./tasks/question_answering)
- [Causal language modeling task guide](./tasks/language_modeling)
- [Masked language modeling task guide](./tasks/masked_language_modeling)
- [Multiple choice task guide](./tasks/multiple_choice)

## ElectraConfig

[[autodoc]] ElectraConfig

## ElectraTokenizer

[[autodoc]] ElectraTokenizer

## ElectraTokenizerFast

[[autodoc]] ElectraTokenizerFast

## Electra specific outputs

[[autodoc]] models.electra.modeling_electra.ElectraForPreTrainingOutput

[[autodoc]] models.electra.modeling_tf_electra.TFElectraForPreTrainingOutput

<frameworkcontent>
<pt>

## ElectraModel

[[autodoc]] ElectraModel
    - forward

## ElectraForPreTraining

[[autodoc]] ElectraForPreTraining
    - forward

## ElectraForCausalLM

[[autodoc]] ElectraForCausalLM
    - forward

## ElectraForMaskedLM

[[autodoc]] ElectraForMaskedLM
    - forward

## ElectraForSequenceClassification

[[autodoc]] ElectraForSequenceClassification
    - forward

## ElectraForMultipleChoice

[[autodoc]] ElectraForMultipleChoice
    - forward

## ElectraForTokenClassification

[[autodoc]] ElectraForTokenClassification
    - forward

## ElectraForQuestionAnswering

[[autodoc]] ElectraForQuestionAnswering
    - forward

</pt>
<tf>

## TFElectraModel

[[autodoc]] TFElectraModel
    - call

## TFElectraForPreTraining

[[autodoc]] TFElectraForPreTraining
    - call

## TFElectraForMaskedLM

[[autodoc]] TFElectraForMaskedLM
    - call

## TFElectraForSequenceClassification

[[autodoc]] TFElectraForSequenceClassification
    - call

## TFElectraForMultipleChoice

[[autodoc]] TFElectraForMultipleChoice
    - call

## TFElectraForTokenClassification

[[autodoc]] TFElectraForTokenClassification
    - call

## TFElectraForQuestionAnswering

[[autodoc]] TFElectraForQuestionAnswering
    - call

</tf>
<jax>

## FlaxElectraModel

[[autodoc]] FlaxElectraModel
    - __call__

## FlaxElectraForPreTraining

[[autodoc]] FlaxElectraForPreTraining
    - __call__

## FlaxElectraForCausalLM

[[autodoc]] FlaxElectraForCausalLM
    - __call__

## FlaxElectraForMaskedLM

[[autodoc]] FlaxElectraForMaskedLM
    - __call__

## FlaxElectraForSequenceClassification

[[autodoc]] FlaxElectraForSequenceClassification
    - __call__

## FlaxElectraForMultipleChoice

[[autodoc]] FlaxElectraForMultipleChoice
    - __call__

## FlaxElectraForTokenClassification

[[autodoc]] FlaxElectraForTokenClassification
    - __call__

## FlaxElectraForQuestionAnswering

[[autodoc]] FlaxElectraForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "c957a68c-8f3f-4d74-a678-cde38acc33dd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dc6b78d56dbe8cd66a97b535e65027e9c2ad324ad709f5de5d7209740a9d7cce", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1e2a2cf7-a9f7-4e73-ba08-f30a77bba942", "node_type": "1", "metadata": {}, "hash": "34ec90fcec75459229a09d392332aa98ceab3b1400c7f1bf6268f54baa0d3797", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoBERTa-PreLayerNorm

## Overview

The RoBERTa-PreLayerNorm model was proposed in [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://arxiv.org/abs/1904.01038) by Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli.
It is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).

The abstract from the paper is the following:

*fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.*

This model was contributed by [andreasmaden](https://huggingface.co/andreasmadsen).
The original code can be found [here](https://github.com/princeton-nlp/DinkyTrain).

## Usage tips

- The implementation is the same as [Roberta](roberta) except instead of using _Add and Norm_ it does _Norm and Add_. _Add_ and _Norm_ refers to the Addition and LayerNormalization as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).
- This is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).

## Resources

- [Text classification task guide](./tasks/sequence_classification)
- [Token classification task guide](./tasks/token_classification)
- [Question answering task guide](./tasks/question_answering)
- [Causal language modeling task guide](./tasks/language_modeling)
- [Masked language modeling task guide](./tasks/masked_language_modeling)
- [Multiple choice task guide](.


key: _node_content
string_value:
{"id_": "1e2a2cf7-a9f7-4e73-ba08-f30a77bba942", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3cd683a7-efa7-4825-b001-0474c1edc100", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dc6b78d56dbe8cd66a97b535e65027e9c2ad324ad709f5de5d7209740a9d7cce", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c957a68c-8f3f-4d74-a678-cde38acc33dd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta-prelayernorm/", "title": "RoBERTa-PreLayerNorm", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7fb50404493f04d8e56ee97b2fa08587d20ab8ba10b14ca4b014b822a464354c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).

The abstract from the paper is the following:

*fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.*

This model was contributed by [andreasmaden](https://huggingface.co/andreasmadsen).
The original code can be found [here](https://github.com/princeton-nlp/DinkyTrain).

## Usage tips

- The implementation is the same as [Roberta](roberta) except instead of using _Add and Norm_ it does _Norm and Add_. _Add_ and _Norm_ refers to the Addition and LayerNormalization as described in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).
- This is identical to using the `--encoder-normalize-before` flag in [fairseq](https://fairseq.readthedocs.io/).

## Resources

- [Text classification task guide](./tasks/sequence_classification)
- [Token classification task guide](./tasks/token_classification)
- [Question answering task guide](./tasks/question_answering)
- [Causal language modeling task guide](./tasks/language_modeling)
- [Masked language modeling task guide](./tasks/masked_language_modeling)
- [Multiple choice task guide](./tasks/multiple_choice)

## RobertaPreLayerNormConfig

[[autodoc]] RobertaPreLayerNormConfig

<frameworkcontent>
<pt>

## RobertaPreLayerNormModel

[[autodoc]] RobertaPreLayerNormModel
    - forward

## RobertaPreLayerNormForCausalLM

[[autodoc]] RobertaPreLayerNormForCausalLM
    - forward

## RobertaPreLayerNormForMaskedLM

[[autodoc]] RobertaPreLayerNormForMaskedLM
    - forward

## RobertaPreLayerNormForSequenceClassification

[[autodoc]] RobertaPreLayerNormForSequenceClassification
    - forward

## RobertaPreLayerNormForMultipleChoice

[[autodoc]] RobertaPreLayerNormForMultipleChoice
    - forward

## RobertaPreLayerNormForTokenClassification

[[autodoc]] RobertaPreLayerNormForTokenClassification
    - forward

## RobertaPreLayerNormForQuestionAnswering

[[autodoc]] RobertaPreLayerNormForQuestionAnswering
    - forward

</pt>
<tf>

## TFRobertaPreLayerNormModel

[[autodoc]] TFRobertaPreLayerNormModel
    - call

## TFRobertaPreLayerNormForCausalLM

[[autodoc]] TFRobertaPreLayerNormForCausalLM
    - call

## TFRobertaPreLayerNormForMaskedLM

[[autodoc]] TFRobertaPreLayerNormForMaskedLM
    - call

## TFRobertaPreLayerNormForSequenceClassification

[[autodoc]] TFRobertaPreLayerNormForSequenceClassification
    - call

## TFRobertaPreLayerNormForMultipleChoice

[[autodoc]] TFRobertaPreLayerNormForMultipleChoice
    - call

## TFRobertaPreLayerNormForTokenClassification

[[autodoc]] TFRobertaPreLayerNormForTokenClassification
    - call

## TFRobertaPreLayerNormForQuestionAnswering

[[autodoc]] TFRobertaPreLayerNormForQuestionAnswering
    - call

</tf>
<jax>

## FlaxRobertaPreLayerNormModel

[[autodoc]] FlaxRobertaPreLayerNormModel
    - __call__

## FlaxRobertaPreLayerNormForCausalLM

[[autodoc]] FlaxRobertaPreLayerNormForCausalLM
    - __call__

## FlaxRobertaPreLayerNormForMaskedLM

[[autodoc]] FlaxRobertaPreLayerNormForMaskedLM
    - __call__

## FlaxRobertaPreLayerNormForSequenceClassification

[[autodoc]] FlaxRobertaPreLayerNormForSequenceClassification
    - __call__

## FlaxRobertaPreLayerNormForMultipleChoice

[[autodoc]] FlaxRobertaPreLayerNormForMultipleChoice
    - __call__

## FlaxRobertaPreLayerNormForTokenClassification

[[autodoc]] FlaxRobertaPreLayerNormForTokenClassification
    - __call__

## FlaxRobertaPreLayerNormForQuestionAnswering

[[autodoc]] FlaxRobertaPreLayerNormForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "b4bf5881-7e3e-4e4f-9b26-c9a8f8e58fcd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilevitv2/", "title": "MobileViTV2", "tokens": 862, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e4a2bd3-9cf4-42e0-aa6a-efadf3efdc9f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilevitv2/", "title": "MobileViTV2", "tokens": 862, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6c89f466767cbf94faa452a3be3239881df0c5571a01e3e81b2abd36f9014f77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2960, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MobileViTV2

## Overview

The MobileViTV2 model was proposed in [Separable Self-attention for Mobile Vision Transformers](https://arxiv.org/abs/2206.02680) by Sachin Mehta and Mohammad Rastegari.

MobileViTV2 is the second version of MobileViT, constructed by replacing the multi-headed self-attention in MobileViT with separable self-attention.

The abstract from the paper is the following:

*Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires O(k2) time complexity with respect to the number of tokens (or patches) k. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. O(k). A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTV2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTV2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running 3.2× faster on a mobile device.*

This model was contributed by [shehan97](https://huggingface.co/shehan97).
The original code can be found [here](https://github.com/apple/ml-cvnets).

## Usage tips

- MobileViTV2 is more like a CNN than a Transformer model. It does not work on sequence data but on batches of images. Unlike ViT, there are no embeddings. The backbone model outputs a feature map.
- One can use [`MobileViTImageProcessor`] to prepare images for the model. Note that if you do your own preprocessing, the pretrained checkpoints expect images to be in BGR pixel order (not RGB).
- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).
- The segmentation model uses a [DeepLabV3](https://arxiv.org/abs/1706.05587) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).

## MobileViTV2Config

[[autodoc]] MobileViTV2Config

## MobileViTV2Model

[[autodoc]] MobileViTV2Model
    - forward

## MobileViTV2ForImageClassification

[[autodoc]] MobileViTV2ForImageClassification
    - forward

## MobileViTV2ForSemanticSegmentation

[[autodoc]] MobileViTV2ForSemanticSegmentation
    - forward


key: _node_content
string_value:
{"id_": "f0bf9c21-a07f-44c3-a5af-c6224d1105e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "41aee283-7d33-4a2c-a3a7-b08749cfa2ca", "node_type": "1", "metadata": {}, "hash": "188b9d991439a2fed573f6a51c522ea5cc660eef77e7d94250fb2192ff3b67c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Grounding DINO

## Overview

The Grounding DINO model was proposed in [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://arxiv.org/abs/2303.05499) by Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang. Grounding DINO extends a closed-set object detection model with a text encoder, enabling open-set object detection. The model achieves remarkable results, such as 52.5 AP on COCO zero-shot.

The abstract from the paper is the following:

*In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a 52.5 AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grouding_dino_architecture.png"
alt="drawing" width="600"/>

<small> Grounding DINO overview. Taken from the <a href="https://arxiv.org/abs/2303.05499">original paper</a>. </small>

This model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).

## Usage tips

- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.
- To separate classes in the text use a period e.g. "a cat. a dog."
- When using multiple classes (e.g. `"a cat. a dog."`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.


key: _node_content
string_value:
{"id_": "41aee283-7d33-4a2c-a3a7-b08749cfa2ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0bf9c21-a07f-44c3-a5af-c6224d1105e2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e44aea0b6b0d14ee59e07350a19a6471ed4ff40f1cf64dc5691bffa177669e61", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d002e1bb-1807-4eb8-a0ab-dc65e2c02634", "node_type": "1", "metadata": {}, "hash": "fef42d10ad6741160fdb6ec00f22268db4231f8acca715345a7600180a4b044d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1225, "end_char_idx": 3373, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a 52.5 AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grouding_dino_architecture.png"
alt="drawing" width="600"/>

<small> Grounding DINO overview. Taken from the <a href="https://arxiv.org/abs/2303.05499">original paper</a>. </small>

This model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).

## Usage tips

- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.
- To separate classes in the text use a period e.g. "a cat. a dog."
- When using multiple classes (e.g. `"a cat. a dog."`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.

Here's how to use the model for zero-shot object detection:

```python
>>> import requests

>>> import torch
>>> from PIL import Image
>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

>>> model_id = "IDEA-Research/grounding-dino-tiny"
>>> device = "cuda"

>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)

>>> image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(image_url, stream=True).raw)
>>> # Check for cats and remote controls
>>> text = "a cat. a remote control."


key: _node_content
string_value:
{"id_": "d002e1bb-1807-4eb8-a0ab-dc65e2c02634", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "41aee283-7d33-4a2c-a3a7-b08749cfa2ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "267bff8f93e9552fb34b955b40655edd4af50f24ae6742278f322cc4e6d1e4c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5d287b7-8f19-4c3d-ab05-07c10e1ff026", "node_type": "1", "metadata": {}, "hash": "38840528246ff7cd0f6775955bb289d6e21230694760df90abdad62daa72bd63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1959, "end_char_idx": 4651, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2303.05499">original paper</a>. </small>

This model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/IDEA-Research/GroundingDINO).

## Usage tips

- One can use [`GroundingDinoProcessor`] to prepare image-text pairs for the model.
- To separate classes in the text use a period e.g. "a cat. a dog."
- When using multiple classes (e.g. `"a cat. a dog."`), use `post_process_grounded_object_detection` from [`GroundingDinoProcessor`] to post process outputs. Since, the labels returned from `post_process_object_detection` represent the indices from the model dimension where prob > threshold.

Here's how to use the model for zero-shot object detection:

```python
>>> import requests

>>> import torch
>>> from PIL import Image
>>> from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection

>>> model_id = "IDEA-Research/grounding-dino-tiny"
>>> device = "cuda"

>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)

>>> image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(image_url, stream=True).raw)
>>> # Check for cats and remote controls
>>> text = "a cat. a remote control."

>>> inputs = processor(images=image, text=text, return_tensors="pt").to(device)
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> results = processor.post_process_grounded_object_detection(
...     outputs,
...     inputs.input_ids,
...     box_threshold=0.4,
...     text_threshold=0.3,
...     target_sizes=[image.size[::-1]]
... )
>>> print(results)
[{'boxes': tensor([[344.6959,  23.1090, 637.1833, 374.2751],
        [ 12.2666,  51.9145, 316.8582, 472.4392],
        [ 38.5742,  70.0015, 176.7838, 118.1806]], device='cuda:0'),
  'labels': ['a cat', 'a cat', 'a remote control'],
  'scores': tensor([0.4785, 0.4381, 0.4776], device='cuda:0')}]
```

## Grounded SAM

One can combine Grounding DINO with the [Segment Anything](sam) model for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) 🌍 for details.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png"
alt="drawing" width="900"/>

<small> Grounded SAM overview.


key: _node_content
string_value:
{"id_": "f5d287b7-8f19-4c3d-ab05-07c10e1ff026", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bf3d26cd-736b-4b34-b57a-9b8dc7a421f3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b318c20228b6d10f08e9e72e3c5a22a6419bcaf0767cbfac2c4ab33a9bb50fc1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d002e1bb-1807-4eb8-a0ab-dc65e2c02634", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/grounding-dino/", "title": "Grounding DINO", "tokens": 1632, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "94806d86f4351f1e84a48fe524911e513215ddf55821f4ed38e0fc0f7ed6a1e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3355, "end_char_idx": 5788, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
a remote control."

>>> inputs = processor(images=image, text=text, return_tensors="pt").to(device)
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> results = processor.post_process_grounded_object_detection(
...     outputs,
...     inputs.input_ids,
...     box_threshold=0.4,
...     text_threshold=0.3,
...     target_sizes=[image.size[::-1]]
... )
>>> print(results)
[{'boxes': tensor([[344.6959,  23.1090, 637.1833, 374.2751],
        [ 12.2666,  51.9145, 316.8582, 472.4392],
        [ 38.5742,  70.0015, 176.7838, 118.1806]], device='cuda:0'),
  'labels': ['a cat', 'a cat', 'a remote control'],
  'scores': tensor([0.4785, 0.4381, 0.4776], device='cuda:0')}]
```

## Grounded SAM

One can combine Grounding DINO with the [Segment Anything](sam) model for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) 🌍 for details.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png"
alt="drawing" width="900"/>

<small> Grounded SAM overview. Taken from the <a href="https://github.com/IDEA-Research/Grounded-Segment-Anything">original repository</a>. </small>

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Grounding DINO. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- Demo notebooks regarding inference with Grounding DINO as well as combining it with [SAM](sam) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Grounding%20DINO). 🌎

## GroundingDinoImageProcessor

[[autodoc]] GroundingDinoImageProcessor
    - preprocess
    - post_process_object_detection

## GroundingDinoProcessor

[[autodoc]] GroundingDinoProcessor
    - post_process_grounded_object_detection

## GroundingDinoConfig

[[autodoc]] GroundingDinoConfig

## GroundingDinoModel

[[autodoc]] GroundingDinoModel
    - forward

## GroundingDinoForObjectDetection

[[autodoc]] GroundingDinoForObjectDetection
    - forward


key: _node_content
string_value:
{"id_": "59a21c2c-156e-44ef-b823-c33252813c55", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d8da0307-f72b-42ed-bfdc-21b4eee253e2", "node_type": "1", "metadata": {}, "hash": "00549dccda6ac18f7182823f17a5f89a9216ec2cf4ea2261346c11eadd9215f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3075, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MarianMT

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=marian">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-marian-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/opus-mt-zh-en">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

A framework for translation models, using the same models as BART. Translations should be similar, but not identical to output in the test set linked to in each model card.
This model was contributed by [sshleifer](https://huggingface.co/sshleifer).


## Implementation Notes

- Each model is about 298 MB on disk, there are more than 1,000 models.
- The list of supported language pairs can be found [here](https://huggingface.co/Helsinki-NLP).
- Models were originally trained by [Jörg Tiedemann](https://researchportal.helsinki.fi/en/persons/j%C3%B6rg-tiedemann) using the [Marian](https://marian-nmt.github.io/) C++ library, which supports fast training and translation.
- All models are transformer encoder-decoders with 6 layers in each component. Each model's performance is documented
  in a model card.
- The 80 opus models that require BPE preprocessing are not supported.
- The modeling code is the same as [`BartForConditionalGeneration`] with a few minor modifications:

  - static (sinusoid) positional embeddings (`MarianConfig.static_position_embeddings=True`)
  - no layernorm_embedding (`MarianConfig.normalize_embedding=False`)
  - the model starts generating with `pad_token_id` (which has 0 as a token_embedding) as the prefix (Bart uses
    `<s/>`),
- Code to bulk convert models can be found in `convert_marian_to_pytorch.py`.


## Naming

- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`
- The language codes used to name models are inconsistent. Two digit codes can usually be found [here](https://developers.google.com/admin-sdk/directory/v1/languages), three digit codes require googling "language
  code {code}".
- Codes formatted like `es_AR` are usually `code_{region}`. That one is Spanish from Argentina.
- The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second
  group use a combination of ISO-639-5 codes and ISO-639-2 codes.


## Examples

- Since Marian models are smaller than many other translation models available in the library, they can be useful for
  fine-tuning experiments and integration tests.
- [Fine-tune on GPU](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/train_distil_marian_enro.sh)

## Multilingual Models

- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`:
- If a model can output multiple languages, and you should specify a language code by prepending the desired output
  language to the `src_text`.
- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).


key: _node_content
string_value:
{"id_": "d8da0307-f72b-42ed-bfdc-21b4eee253e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59a21c2c-156e-44ef-b823-c33252813c55", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e400a98596cb7dc08da79759c944cd2c09fe4f1ac0e692823c3cc3db870de945", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3414972-83a8-4973-b0a4-08c6cd9ecb33", "node_type": "1", "metadata": {}, "hash": "b763b721895cd79422aa777b710922e81e61a50d45b95a0986078267d9497f71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1739, "end_char_idx": 4290, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Naming

- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`
- The language codes used to name models are inconsistent. Two digit codes can usually be found [here](https://developers.google.com/admin-sdk/directory/v1/languages), three digit codes require googling "language
  code {code}".
- Codes formatted like `es_AR` are usually `code_{region}`. That one is Spanish from Argentina.
- The models were converted in two stages. The first 1000 models use ISO-639-2 codes to identify languages, the second
  group use a combination of ISO-639-5 codes and ISO-639-2 codes.


## Examples

- Since Marian models are smaller than many other translation models available in the library, they can be useful for
  fine-tuning experiments and integration tests.
- [Fine-tune on GPU](https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/train_distil_marian_enro.sh)

## Multilingual Models

- All model names use the following format: `Helsinki-NLP/opus-mt-{src}-{tgt}`:
- If a model can output multiple languages, and you should specify a language code by prepending the desired output
  language to the `src_text`.
- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).
- Note that if a model is only multilingual on the source side, like `Helsinki-NLP/opus-mt-roa-en`, no language
  codes are required.

New multi-lingual models from the [Tatoeba-Challenge repo](https://github.com/Helsinki-NLP/Tatoeba-Challenge)
require 3 character language codes:

```python
>>> from transformers import MarianMTModel, MarianTokenizer

>>> src_text = [
...     ">>fra<< this is a sentence in english that we want to translate to french",
...     ">>por<< This should go to portuguese",
...     ">>esp<< And this to Spanish",
... ]

>>> model_name = "Helsinki-NLP/opus-mt-en-roa"
>>> tokenizer = MarianTokenizer.from_pretrained(model_name)
>>> print(tokenizer.supported_language_codes)
['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']

>>> model = MarianMTModel.from_pretrained(model_name)
>>> translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
>>> [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
["c'est une phrase en anglais que nous voulons traduire en français",
 'Isto deve ir para o português.


key: _node_content
string_value:
{"id_": "e3414972-83a8-4973-b0a4-08c6cd9ecb33", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d8da0307-f72b-42ed-bfdc-21b4eee253e2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "266b996c329a4c9e7af1e40d698f4d5d325b453fcf9942581115b95ba78e3f79", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "063bd867-5a5c-4ea6-b607-60aef33e2eb2", "node_type": "1", "metadata": {}, "hash": "e9e8e09492690d1c849851363e67b89752b7278f8fc4d307fe4fff7848cbf4a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2902, "end_char_idx": 6221, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- You can see a models's supported language codes in its model card, under target constituents, like in [opus-mt-en-roa](https://huggingface.co/Helsinki-NLP/opus-mt-en-roa).
- Note that if a model is only multilingual on the source side, like `Helsinki-NLP/opus-mt-roa-en`, no language
  codes are required.

New multi-lingual models from the [Tatoeba-Challenge repo](https://github.com/Helsinki-NLP/Tatoeba-Challenge)
require 3 character language codes:

```python
>>> from transformers import MarianMTModel, MarianTokenizer

>>> src_text = [
...     ">>fra<< this is a sentence in english that we want to translate to french",
...     ">>por<< This should go to portuguese",
...     ">>esp<< And this to Spanish",
... ]

>>> model_name = "Helsinki-NLP/opus-mt-en-roa"
>>> tokenizer = MarianTokenizer.from_pretrained(model_name)
>>> print(tokenizer.supported_language_codes)
['>>zlm_Latn<<', '>>mfe<<', '>>hat<<', '>>pap<<', '>>ast<<', '>>cat<<', '>>ind<<', '>>glg<<', '>>wln<<', '>>spa<<', '>>fra<<', '>>ron<<', '>>por<<', '>>ita<<', '>>oci<<', '>>arg<<', '>>min<<']

>>> model = MarianMTModel.from_pretrained(model_name)
>>> translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
>>> [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
["c'est une phrase en anglais que nous voulons traduire en français",
 'Isto deve ir para o português.',
 'Y esto al español']
```

Here is the code to see all available pretrained models on the hub:

```python
from huggingface_hub import list_models

model_list = list_models()
org = "Helsinki-NLP"
model_ids = [x.id for x in model_list if x.id.startswith(org)]
suffix = [x.split("/")[1] for x in model_ids]
old_style_multi_models = [f"{org}/{s}" for s in suffix if s != s.lower()]
```

## Old Style Multi-Lingual Models

These are the old style multi-lingual models ported from the OPUS-MT-Train repo: and the members of each language
group:

```python no-style
['Helsinki-NLP/opus-mt-NORTH_EU-NORTH_EU',
 'Helsinki-NLP/opus-mt-ROMANCE-en',
 'Helsinki-NLP/opus-mt-SCANDINAVIA-SCANDINAVIA',
 'Helsinki-NLP/opus-mt-de-ZH',
 'Helsinki-NLP/opus-mt-en-CELTIC',
 'Helsinki-NLP/opus-mt-en-ROMANCE',
 'Helsinki-NLP/opus-mt-es-NORWAY',
 'Helsinki-NLP/opus-mt-fi-NORWAY',
 'Helsinki-NLP/opus-mt-fi-ZH',
 'Helsinki-NLP/opus-mt-fi_nb_no_nn_ru_sv_en-SAMI',
 'Helsinki-NLP/opus-mt-sv-NORWAY',
 'Helsinki-NLP/opus-mt-sv-ZH']
GROUP_MEMBERS = {
 'ZH': ['cmn', 'cn', 'yue', 'ze_zh', 'zh_cn', 'zh_CN', 'zh_HK', 'zh_tw', 'zh_TW', 'zh_yue', 'zhs', 'zht', 'zh'],
 'ROMANCE': ['fr', 'fr_BE', 'fr_CA', 'fr_FR', 'wa', 'frp', 'oc', 'ca', 'rm', 'lld', 'fur', 'lij', 'lmo', 'es', 'es_AR', 'es_CL', 'es_CO', 'es_CR', 'es_DO', 'es_EC', 'es_ES', 'es_GT', 'es_HN', 'es_MX', 'es_NI', 'es_PA', 'es_PE', 'es_PR', 'es_SV', 'es_UY', 'es_VE', 'pt', 'pt_br', 'pt_BR', 'pt_PT', 'gl', 'lad', 'an', 'mwl', 'it', 'it_IT', 'co', 'nap', 'scn', 'vec', 'sc', 'ro', 'la'],
 'NORTH_EU': ['de', 'nl', 'fy', 'af', 'da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],
 'SCANDINAVIA': ['da', 'fo', 'is', 'no', 'nb', 'nn', 'sv'],
 'SAMI': ['se', 'sma', 'smj', 'smn', 'sms'],
 'NORWAY': ['nb_NO', 'nb', 'nn_NO', 'nn', 'nog', 'no_nb', 'no'],
 'CELTIC': ['ga', 'cy', 'br', 'gd', 'kw', 'gv']
}
```

Example of translating english to many romance languages, using old-style 2 character language codes


key: _node_content
string_value:
{"id_": "063bd867-5a5c-4ea6-b607-60aef33e2eb2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2567f576-ca06-4dee-8abb-fc80b346a8b1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fae59e5fedda3ee339831bf7dda4f082d89e7b573230fc361cf09ae6b25f090a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3414972-83a8-4973-b0a4-08c6cd9ecb33", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/marian/", "title": "MarianMT", "tokens": 2443, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "299b371787cd2a53ecb797da527760a513aeb04bc2caae36567f9227667b2fb8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6224, "end_char_idx": 7781, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> from transformers import MarianMTModel, MarianTokenizer

>>> src_text = [
...     ">>fr<< this is a sentence in english that we want to translate to french",
...     ">>pt<< This should go to portuguese",
...     ">>es<< And this to Spanish",
... ]

>>> model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
>>> tokenizer = MarianTokenizer.from_pretrained(model_name)

>>> model = MarianMTModel.from_pretrained(model_name)
>>> translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True))
>>> tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
["c'est une phrase en anglais que nous voulons traduire en français", 
 'Isto deve ir para o português.',
 'Y esto al español']
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)
- [Causal language modeling task guide](../tasks/language_modeling)

## MarianConfig

[[autodoc]] MarianConfig

## MarianTokenizer

[[autodoc]] MarianTokenizer
    - build_inputs_with_special_tokens

<frameworkcontent>
<pt>

## MarianModel

[[autodoc]] MarianModel
    - forward

## MarianMTModel

[[autodoc]] MarianMTModel
    - forward

## MarianForCausalLM

[[autodoc]] MarianForCausalLM
    - forward

</pt>
<tf>

## TFMarianModel

[[autodoc]] TFMarianModel
    - call

## TFMarianMTModel

[[autodoc]] TFMarianMTModel
    - call

</tf>
<jax>

## FlaxMarianModel

[[autodoc]] FlaxMarianModel
    - __call__

## FlaxMarianMTModel

[[autodoc]] FlaxMarianMTModel
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "2146b1b6-0759-4a1a-8ad1-fb3dc684c7b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder/", "title": "VisionTextDualEncoder", "tokens": 543, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b089cfeb-af35-46d7-ab6d-3a99496e51de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-text-dual-encoder/", "title": "VisionTextDualEncoder", "tokens": 543, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b242f89312a2c8b5841215ec6410d9e9a0f338958a2c9fed9df342174e57894b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1579, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VisionTextDualEncoder

## Overview

The [`VisionTextDualEncoderModel`] can be used to initialize a vision-text dual encoder model with
any pretrained vision autoencoding model as the vision encoder (*e.g.* [ViT](vit), [BEiT](beit), [DeiT](deit)) and any pretrained text autoencoding model as the text encoder (*e.g.* [RoBERTa](roberta), [BERT](bert)). Two projection layers are added on top of both the vision and text encoder to project the output embeddings
to a shared latent space. The projection layers are randomly initialized so the model should be fine-tuned on a
downstream task. This model can be used to align the vision-text embeddings using CLIP like contrastive image-text
training and then can be used for zero-shot vision tasks such image-classification or retrieval.

In [LiT: Zero-Shot Transfer with Locked-image Text Tuning](https://arxiv.org/abs/2111.07991) it is shown how
leveraging pre-trained (locked/frozen) image and text model for contrastive learning yields significant improvement on
new zero-shot vision tasks such as image classification or retrieval.

## VisionTextDualEncoderConfig

[[autodoc]] VisionTextDualEncoderConfig

## VisionTextDualEncoderProcessor

[[autodoc]] VisionTextDualEncoderProcessor

<frameworkcontent>
<pt>

## VisionTextDualEncoderModel

[[autodoc]] VisionTextDualEncoderModel
    - forward

</pt>
<tf>

## FlaxVisionTextDualEncoderModel

[[autodoc]] FlaxVisionTextDualEncoderModel
    - __call__

</tf>
<jax>

## TFVisionTextDualEncoderModel

[[autodoc]] TFVisionTextDualEncoderModel
    - call

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "cf36e9b2-e601-4633-a1f3-b79079bb25d8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "748c059e-f831-4e36-94bf-5a412230141e", "node_type": "1", "metadata": {}, "hash": "2d00388786c48a4f235b7d2e610f12909bbbfd51ff605d8952ed50a732463673", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# NLLB-MOE


## Overview

The NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-jussà, James Cross, Onur Çelebi,
Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,
Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,
Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Safiyyah Saleem, Holger Schwenk, and Jeff Wang.

The abstract of the paper is the following:

*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the
200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by
first contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed
at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of
Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training
improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using
a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.
Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).
The original code can be found [here](https://github.com/facebookresearch/fairseq).

## Usage tips

- M2M100ForConditionalGeneration is the base model for both NLLB and NLLB MoE
- The NLLB-MoE is very similar to the NLLB model, but it's feed forward layer is based on the implementation of SwitchTransformers.
- The tokenizer is the same as the NLLB models.

## Implementation differences with SwitchTransformers

The biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the 
highest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, 
which means that tokens have less probability of being forwarded.


key: _node_content
string_value:
{"id_": "748c059e-f831-4e36-94bf-5a412230141e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf36e9b2-e601-4633-a1f3-b79079bb25d8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "783ee40db1474b31e5b2cc41c13efd257e2d92d7d770f70e1557da5733348484", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6739a23-44e1-45b4-9f5e-26ad20d9d80a", "node_type": "1", "metadata": {}, "hash": "385c780de5a64533d87c0100236afa3bd959ff47ba979011d2e4c1060ad2c756", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1457, "end_char_idx": 4426, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then, we created datasets and models aimed
at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of
Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training
improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using
a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.
Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ).
The original code can be found [here](https://github.com/facebookresearch/fairseq).

## Usage tips

- M2M100ForConditionalGeneration is the base model for both NLLB and NLLB MoE
- The NLLB-MoE is very similar to the NLLB model, but it's feed forward layer is based on the implementation of SwitchTransformers.
- The tokenizer is the same as the NLLB models.

## Implementation differences with SwitchTransformers

The biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the 
highest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, 
which means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, `SwitchTransformers` still adds its unmodified hidden 
states (kind of like a residual connection) while they are masked in `NLLB`'s top-2 routing mechanism. 

## Generating with NLLB-MoE

The available checkpoints require around 350GB of storage. Make sure to use `accelerate` if you do not have enough RAM on your machine.

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-moe-54b")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-moe-54b")

>>> article = "Previously, Ring's CEO, Jamie Siminoff, remarked the company started when his doorbell wasn't audible from his shop in his garage."


key: _node_content
string_value:
{"id_": "a6739a23-44e1-45b4-9f5e-26ad20d9d80a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "748c059e-f831-4e36-94bf-5a412230141e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c05da317b8ca57078db2d04d927f6b454194b7d397c63b00ee99afe0d471a061", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2cb1e5b-c41f-4768-bd67-09569e8dd91c", "node_type": "1", "metadata": {}, "hash": "3f3759b436ce7e9a0576462292acf4dece303bc24ae4a08e06a544a7e6645a48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2792, "end_char_idx": 5943, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Implementation differences with SwitchTransformers

The biggest difference is the way the tokens are routed. NLLB-MoE uses a `top-2-gate` which means that for each input, only the top two experts are selected based on the 
highest predicted probabilities from the gating network, and the remaining experts are ignored. In `SwitchTransformers`, only the top-1 probabilities are computed, 
which means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, `SwitchTransformers` still adds its unmodified hidden 
states (kind of like a residual connection) while they are masked in `NLLB`'s top-2 routing mechanism. 

## Generating with NLLB-MoE

The available checkpoints require around 350GB of storage. Make sure to use `accelerate` if you do not have enough RAM on your machine.

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-moe-54b")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-moe-54b")

>>> article = "Previously, Ring's CEO, Jamie Siminoff, remarked the company started when his doorbell wasn't audible from his shop in his garage."
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["fra_Latn"], max_length=50
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"Auparavant, le PDG de Ring, Jamie Siminoff, a fait remarquer que la société avait commencé lorsque sa sonnette n'était pas audible depuis son magasin dans son garage."
```

### Generating from any other language than English

English (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.

See example below for a translation from romanian to german:

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-moe-54b", src_lang="ron_Latn")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-moe-54b")

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)


key: _node_content
string_value:
{"id_": "e2cb1e5b-c41f-4768-bd67-09569e8dd91c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f92b581-716a-4047-8f46-1fa6d7e256e6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a37bb9a3daf60c76130a430c85ceb53fb567c2c1575d836c4a74949e3a9697e6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6739a23-44e1-45b4-9f5e-26ad20d9d80a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb-moe/", "title": "NLLB-MOE", "tokens": 1704, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c7a8da6f91bd1faffa2b2d9691d922c30c6e837d1c556f6f504271f9221f8ab3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4427, "end_char_idx": 6292, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["fra_Latn"], max_length=50
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"Auparavant, le PDG de Ring, Jamie Siminoff, a fait remarquer que la société avait commencé lorsque sa sonnette n'était pas audible depuis son magasin dans son garage."
```

### Generating from any other language than English

English (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.

See example below for a translation from romanian to german:

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-moe-54b", src_lang="ron_Latn")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-moe-54b")

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)


## NllbMoeConfig

[[autodoc]] NllbMoeConfig

## NllbMoeTop2Router

[[autodoc]] NllbMoeTop2Router
    - route_tokens
    - forward

## NllbMoeSparseMLP

[[autodoc]] NllbMoeSparseMLP
    - forward

## NllbMoeModel

[[autodoc]] NllbMoeModel
    - forward

## NllbMoeForConditionalGeneration

[[autodoc]] NllbMoeForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "20c7be2e-f7e3-42a8-94e4-1b9ecce77daa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e0003566b09b76bb461a72f902c9cf83e4b9dad4380a2130fecafc6214ca2a00", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f115a5dd-8032-4326-aafa-97ee59374701", "node_type": "1", "metadata": {}, "hash": "e1e69a561b5a68cd927ec376477ee5d56399eefd95e3659b365cd237db801918", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2702, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPTSAN-japanese

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The GPTSAN-japanese model was released in the repository by Toshiyuki Sakamoto (tanreinama).

GPTSAN is a Japanese language model using Switch Transformer. It has the same structure as the model introduced as Prefix LM
in the T5 paper, and support both Text Generation and Masked Language Modeling tasks. These basic tasks similarly can
fine-tune for translation or summarization.

### Usage example

The `generate()` method can be used to generate text using GPTSAN-Japanese model.

```python
>>> from transformers import AutoModel, AutoTokenizer
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("Tanrei/GPTSAN-japanese")
>>> model = AutoModel.from_pretrained("Tanrei/GPTSAN-japanese").cuda()
>>> x_tok = tokenizer("は、", prefix_text="織田信長", return_tensors="pt")
>>> torch.manual_seed(0)
>>> gen_tok = model.generate(x_tok.input_ids.cuda(), token_type_ids=x_tok.token_type_ids.cuda(), max_new_tokens=20)
>>> tokenizer.decode(gen_tok[0])
'織田信長は、2004年に『戦国BASARA』のために、豊臣秀吉'
```

## GPTSAN Features

GPTSAN has some unique features. It has a model structure of Prefix-LM. It works as a shifted Masked Language Model for Prefix Input tokens. Un-prefixed inputs behave like normal generative models.
The Spout vector is a GPTSAN specific input. Spout is pre-trained with random inputs, but you can specify a class of text or an arbitrary vector during fine-tuning. This allows you to indicate the tendency of the generated text.
GPTSAN has a sparse Feed Forward based on Switch-Transformer. You can also add other layers and train them partially. See the original GPTSAN repository for details.

### Prefix-LM Model

GPTSAN has the structure of the model named Prefix-LM in the `T5` paper. (The original GPTSAN repository calls it `hybrid`)
In GPTSAN, the `Prefix` part of Prefix-LM, that is, the input position that can be referenced by both tokens, can be specified with any length.
Arbitrary lengths can also be specified differently for each batch.
This length applies to the text entered in `prefix_text` for the tokenizer.
The tokenizer returns the mask of the `Prefix` part of Prefix-LM as `token_type_ids`.
The model treats the part where `token_type_ids` is 1 as a `Prefix` part, that is, the input can refer to both tokens before and after.

## Usage tips

Specifying the Prefix part is done with a mask passed to self-attention.


key: _node_content
string_value:
{"id_": "f115a5dd-8032-4326-aafa-97ee59374701", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e0003566b09b76bb461a72f902c9cf83e4b9dad4380a2130fecafc6214ca2a00", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "20c7be2e-f7e3-42a8-94e4-1b9ecce77daa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bc4b08b4a3eaf147f779260cc67442ca53d32545b1542e157747f8a08746313b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92bbbe8d-9658-45ca-a95b-dc2041e8d003", "node_type": "1", "metadata": {}, "hash": "74a89b65a6c9a6dcf31a94c0d7d0c90b94e4a22402997d946631154377c4088c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1398, "end_char_idx": 3695, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It has a model structure of Prefix-LM. It works as a shifted Masked Language Model for Prefix Input tokens. Un-prefixed inputs behave like normal generative models.
The Spout vector is a GPTSAN specific input. Spout is pre-trained with random inputs, but you can specify a class of text or an arbitrary vector during fine-tuning. This allows you to indicate the tendency of the generated text.
GPTSAN has a sparse Feed Forward based on Switch-Transformer. You can also add other layers and train them partially. See the original GPTSAN repository for details.

### Prefix-LM Model

GPTSAN has the structure of the model named Prefix-LM in the `T5` paper. (The original GPTSAN repository calls it `hybrid`)
In GPTSAN, the `Prefix` part of Prefix-LM, that is, the input position that can be referenced by both tokens, can be specified with any length.
Arbitrary lengths can also be specified differently for each batch.
This length applies to the text entered in `prefix_text` for the tokenizer.
The tokenizer returns the mask of the `Prefix` part of Prefix-LM as `token_type_ids`.
The model treats the part where `token_type_ids` is 1 as a `Prefix` part, that is, the input can refer to both tokens before and after.

## Usage tips

Specifying the Prefix part is done with a mask passed to self-attention.
When token_type_ids=None or all zero, it is equivalent to regular causal mask

for example:

>>> x_token = tokenizer("ｱｲｳｴ")
input_ids:      | SOT | SEG | ｱ | ｲ | ｳ | ｴ |
token_type_ids: | 1   | 0   | 0 | 0 | 0 | 0 |
prefix_lm_mask:
SOT | 1 0 0 0 0 0 |
SEG | 1 1 0 0 0 0 |
ｱ   | 1 1 1 0 0 0 |
ｲ   | 1 1 1 1 0 0 |
ｳ   | 1 1 1 1 1 0 |
ｴ   | 1 1 1 1 1 1 |

>>> x_token = tokenizer("", prefix_text="ｱｲｳｴ")
input_ids:      | SOT | ｱ | ｲ | ｳ | ｴ | SEG |
token_type_ids: | 1   | 1 | 1 | 1 | 1 | 0  |
prefix_lm_mask:
SOT | 1 1 1 1 1 0 |
ｱ   | 1 1 1 1 1 0 |
ｲ   | 1 1 1 1 1 0 |
ｳ   | 1 1 1 1 1 0 |
ｴ   | 1 1 1 1 1 0 |
SEG | 1 1 1 1 1 1 |

>>> x_token = tokenizer("ｳｴ", prefix_text="ｱｲ")
input_ids:      | SOT | ｱ | ｲ | SEG | ｳ | ｴ |
token_type_ids: | 1   | 1 | 1 | 0   | 0 | 0 |
prefix_lm_mask:
SOT | 1 1 1 0 0 0 |
ｱ   | 1 1 1 0 0 0 |
ｲ   | 1 1 1 0 0 0 |
SEG | 1 1 1 1 0 0 |
ｳ   | 1 1 1 1 1 0 |
ｴ   | 1 1 1 1 1 1 |

### Spout Vector

A Spout Vector is a special vector for controlling text generation.


key: _node_content
string_value:
{"id_": "92bbbe8d-9658-45ca-a95b-dc2041e8d003", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "65c2649f-0dcc-4be9-b3ce-d69c5288c992", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e0003566b09b76bb461a72f902c9cf83e4b9dad4380a2130fecafc6214ca2a00", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f115a5dd-8032-4326-aafa-97ee59374701", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gptsan-japanese/", "title": "GPTSAN-japanese", "tokens": 1558, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "98e11795cb2b3716948b3433f08caa114c0dbff55807a53f1978b38bd24aeee9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3696, "end_char_idx": 4437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This vector is treated as the first embedding in self-attention to bring extraneous attention to the generated tokens.
In the pre-trained model published from `Tanrei/GPTSAN-japanese`, the Spout Vector is a 128-dimensional vector that passes through 8 fully connected layers in the model and is projected into the space acting as external attention.
The Spout Vector projected by the fully connected layer is split to be passed to all self-attentions.

## GPTSanJapaneseConfig

[[autodoc]] GPTSanJapaneseConfig

## GPTSanJapaneseTokenizer

[[autodoc]] GPTSanJapaneseTokenizer

## GPTSanJapaneseModel

[[autodoc]] GPTSanJapaneseModel

## GPTSanJapaneseForConditionalGeneration

[[autodoc]] GPTSanJapaneseForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "71fee20a-7fdc-4dcc-b9ff-c53b69edecba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c5c40e9deff78bc8abcfdcbcd6450a7b1467e7270b619977c3e9e99a03002693", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f4730f8e-75f0-458f-a2a5-aa343379ac30", "node_type": "1", "metadata": {}, "hash": "d30e68a883caf4322ea4140dd9e352882be2759bc9b73a03abb8af50571ea3c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Neighborhood Attention Transformer

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

NAT was proposed in [Neighborhood Attention Transformer](https://arxiv.org/abs/2204.07143)
by Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi.

It is a hierarchical vision transformer based on Neighborhood Attention, a sliding-window self attention pattern.

The abstract from the paper is the following:

*We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision.
NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a
linear time and space complexity compared to the quadratic complexity of SA. The sliding-window pattern allows NA's
receptive field to grow without needing extra pixel shifts, and preserves translational equivariance, unlike
Swin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood Attention Extension), a Python package
with efficient C++ and CUDA kernels, which allows NA to run up to 40% faster than Swin's WSA while using up to 25% less
memory. We further present Neighborhood Attention Transformer (NAT), a new hierarchical transformer design based on NA
that boosts image classification and downstream vision performance. Experimental results on NAT are competitive;
NAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MS-COCO and 48.4% mIoU on ADE20K, which is 1.9%
ImageNet accuracy, 1.0% COCO mAP, and 2.6% ADE20K mIoU improvement over a Swin model with similar size. *

<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/neighborhood-attention-pattern.jpg"
alt="drawing" width="600"/>

<small> Neighborhood Attention compared to other attention patterns.
Taken from the <a href="https://arxiv.org/abs/2204.07143">original paper</a>.</small>

This model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).
The original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).

## Usage tips

- One can use the [`AutoImageProcessor`] API to prepare images for the model.
- NAT can be used as a *backbone*. When `output_hidden_states = True`,
it will output both `hidden_states` and `reshaped_hidden_states`.
The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than
`(batch_size, height, width, num_channels)`.

Notes:
- NAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention.
You can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten),
or build on your system by running `pip install natten`.
Note that the latter will likely take time to compile. NATTEN does not support Windows devices yet.
- Patch size of 4 is only supported at the moment.


key: _node_content
string_value:
{"id_": "f4730f8e-75f0-458f-a2a5-aa343379ac30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e99cf0ee-37a9-470c-a7d6-de698838be2d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c5c40e9deff78bc8abcfdcbcd6450a7b1467e7270b619977c3e9e99a03002693", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71fee20a-7fdc-4dcc-b9ff-c53b69edecba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nat/", "title": "Neighborhood Attention Transformer", "tokens": 1124, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0ea7a7acef97b69fe28f9b3f02f2d154e4063b1684c5ba8d2149067bad7974c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1849, "end_char_idx": 4093, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

<img
src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/neighborhood-attention-pattern.jpg"
alt="drawing" width="600"/>

<small> Neighborhood Attention compared to other attention patterns.
Taken from the <a href="https://arxiv.org/abs/2204.07143">original paper</a>.</small>

This model was contributed by [Ali Hassani](https://huggingface.co/alihassanijr).
The original code can be found [here](https://github.com/SHI-Labs/Neighborhood-Attention-Transformer).

## Usage tips

- One can use the [`AutoImageProcessor`] API to prepare images for the model.
- NAT can be used as a *backbone*. When `output_hidden_states = True`,
it will output both `hidden_states` and `reshaped_hidden_states`.
The `reshaped_hidden_states` have a shape of `(batch, num_channels, height, width)` rather than
`(batch_size, height, width, num_channels)`.

Notes:
- NAT depends on [NATTEN](https://github.com/SHI-Labs/NATTEN/)'s implementation of Neighborhood Attention.
You can install it with pre-built wheels for Linux by referring to [shi-labs.com/natten](https://shi-labs.com/natten),
or build on your system by running `pip install natten`.
Note that the latter will likely take time to compile. NATTEN does not support Windows devices yet.
- Patch size of 4 is only supported at the moment.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with NAT.

<PipelineTag pipeline="image-classification"/>

- [`NatForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## NatConfig

[[autodoc]] NatConfig

## NatModel

[[autodoc]] NatModel
    - forward

## NatForImageClassification

[[autodoc]] NatForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "a5b89ce6-b69e-46e2-ac8d-9c969c53b9c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cdece756-509c-426d-b48a-b086a295b00c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a7534d64910fb25dc679e8a2abc0a3bdf49eeb539b4efaa54583429adf616634", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "208ea302-a92b-4a81-9380-99e43747148e", "node_type": "1", "metadata": {}, "hash": "3a1931a4ab0ad52105f36450058ff76de69a3bc0de520be2f82a5aeb0f9dc149", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2191, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Depth Anything V2

## Overview

Depth Anything V2 was introduced in [the paper of the same name](https://arxiv.org/abs/2406.09414) by Lihe Yang et al. It uses the same architecture as the original [Depth Anything model](depth_anything), but uses synthetic data and a larger capacity teacher model to achieve much finer and robust depth predictions.

The abstract from the paper is the following:

*This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg"
alt="drawing" width="600"/>

<small> Depth Anything overview. Taken from the <a href="https://arxiv.org/abs/2401.10891">original paper</a>.</small>

The Depth Anything models were contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/DepthAnything/Depth-Anything-V2).

## Usage example

There are 2 main ways to use Depth Anything V2: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself.


key: _node_content
string_value:
{"id_": "208ea302-a92b-4a81-9380-99e43747148e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cdece756-509c-426d-b48a-b086a295b00c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a7534d64910fb25dc679e8a2abc0a3bdf49eeb539b4efaa54583429adf616634", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5b89ce6-b69e-46e2-ac8d-9c969c53b9c8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "87cc6e947a538a994b9457976f50ad4a8c311ef7510a8748051f5d319ca6c5bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "09d454a1-27c9-43de-baaa-6de06d9bc27f", "node_type": "1", "metadata": {}, "hash": "8e9c07136b9189b0ce9b5900b21ed8233444d62bb311fc64bd3b68152e377a8a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 438, "end_char_idx": 4037, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/depth_anything_overview.jpg"
alt="drawing" width="600"/>

<small> Depth Anything overview. Taken from the <a href="https://arxiv.org/abs/2401.10891">original paper</a>.</small>

The Depth Anything models were contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/DepthAnything/Depth-Anything-V2).

## Usage example

There are 2 main ways to use Depth Anything V2: either using the pipeline API, which abstracts away all the complexity for you, or by using the `DepthAnythingForDepthEstimation` class yourself.

### Pipeline API

The pipeline allows to use the model in a few lines of code:

```python
>>> from transformers import pipeline
>>> from PIL import Image
>>> import requests

>>> # load pipe
>>> pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")

>>> # load image
>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> # inference
>>> depth = pipe(image)["depth"]
```

### Using the model yourself

If you want to do the pre- and post-processing yourself, here's how to do that:

```python
>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation
>>> import torch
>>> import numpy as np
>>> from PIL import Image
>>> import requests

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image_processor = AutoImageProcessor.from_pretrained("depth-anything/Depth-Anything-V2-Small-hf")
>>> model = AutoModelForDepthEstimation.from_pretrained("depth-anything/Depth-Anything-V2-Small-hf")

>>> # prepare image for the model
>>> inputs = image_processor(images=image, return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)
...     predicted_depth = outputs.predicted_depth

>>> # interpolate to original size
>>> prediction = torch.nn.functional.interpolate(
...     predicted_depth.unsqueeze(1),
...     size=image.size[::-1],
...     mode="bicubic",
...     align_corners=False,
... )

>>> # visualize the prediction
>>> output = prediction.squeeze().cpu().numpy()
>>> formatted = (output * 255 / np.max(output)).astype("uint8")
>>> depth = Image.fromarray(formatted)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Depth Anything.


key: _node_content
string_value:
{"id_": "09d454a1-27c9-43de-baaa-6de06d9bc27f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cdece756-509c-426d-b48a-b086a295b00c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a7534d64910fb25dc679e8a2abc0a3bdf49eeb539b4efaa54583429adf616634", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "208ea302-a92b-4a81-9380-99e43747148e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/depth_anything_v2/", "title": "Depth Anything V2", "tokens": 1299, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8d227c962314472402aedddcf385a321a037b100b87f26f20708a0a1a87f78b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4039, "end_char_idx": 4948, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Monocular depth estimation task guide](../tasks/depth_estimation)
- [Depth Anything V2 demo](https://huggingface.co/spaces/depth-anything/Depth-Anything-V2).
- A notebook showcasing inference with [`DepthAnythingForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Depth%20Anything/Predicting_depth_in_an_image_with_Depth_Anything.ipynb). 🌎
- [Core ML conversion of the `small` variant for use on Apple Silicon](https://huggingface.co/apple/coreml-depth-anything-v2-small).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DepthAnythingConfig

[[autodoc]] DepthAnythingConfig

## DepthAnythingForDepthEstimation

[[autodoc]] DepthAnythingForDepthEstimation
    - forward


key: _node_content
string_value:
{"id_": "e8934f6f-3459-4b7f-bebe-37606ddd604c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "27c54315-fc4d-42c7-afc8-222f7f71dd1b", "node_type": "1", "metadata": {}, "hash": "22770f56cc9796afcdbcaae6a9181fedc64997e178a6675226f432c5413d9aa4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3419, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ALBERT

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=albert">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-albert-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/albert-base-v2">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The ALBERT model was proposed in [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942) by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma,
Radu Soricut. It presents two parameter-reduction techniques to lower memory consumption and increase the training
speed of BERT:

- Splitting the embedding matrix into two smaller matrices.
- Using repeating layers split among groups.

The abstract from the paper is the following:

*Increasing model size when pretraining natural language representations often results in improved performance on
downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations,
longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction
techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows
that our proposed methods lead to models that scale much better compared to the original BERT. We also use a
self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks
with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and
SQuAD benchmarks while having fewer parameters compared to BERT-large.*

This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).

## Usage tips

- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather
  than the left.
- ALBERT uses repeating layers which results in a small memory footprint, however the computational cost remains
  similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same
  number of (repeating) layers.
- Embedding size E is different from hidden size H justified because the embeddings are context independent (one embedding vector represents one token), whereas hidden states are context dependent (one hidden state represents a sequence of tokens) so it's more logical to have H >> E. Also, the embedding matrix is large since it's V x E (V being the vocab size). If E < H, it has less parameters.
- Layers are split in groups that share parameters (to save memory).
Next sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have been swapped or not.



This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).


## Resources


key: _node_content
string_value:
{"id_": "27c54315-fc4d-42c7-afc8-222f7f71dd1b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8934f6f-3459-4b7f-bebe-37606ddd604c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "05c9d855cec923846575ddb5535c20fe9d06c2ce9d8d2bb381820cee229c4a09", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "314949c2-03c8-4ff5-9d64-e296d92c5c6e", "node_type": "1", "metadata": {}, "hash": "3313f2db2e48fc4ccefa876670b462a06683db2724d3bc03cf6d0da0c79e8619", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3159, "end_char_idx": 6491, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).


## Resources


The resources provided in the following sections consist of a list of official Hugging Face and community (indicated by 🌎) resources to help you get started with AlBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


<PipelineTag pipeline="text-classification"/>


- [`AlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification).


- [`TFAlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification).

- [`FlaxAlbertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- Check the [Text classification task guide](../tasks/sequence_classification) on how to use the model.


<PipelineTag pipeline="token-classification"/>


- [`AlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification).


- [`TFAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).



- [`FlaxAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Token classification task guide](../tasks/token_classification) on how to use the model.

<PipelineTag pipeline="fill-mask"/>

- [`AlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).


key: _node_content
string_value:
{"id_": "314949c2-03c8-4ff5-9d64-e296d92c5c6e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "27c54315-fc4d-42c7-afc8-222f7f71dd1b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6360e5d37b11bb2eaf6c4b95338eef0f84befb4a0e92247255dba64ae8c5b146", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12fb498d-2da7-4e2f-ae6f-6e7f4d763b07", "node_type": "1", "metadata": {}, "hash": "9c5b42ddac3b7a3e3e0ac1ebc0874e038544a2735244f9adbb3832ba708c9736", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4838, "end_char_idx": 8013, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).



- [`FlaxAlbertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Token classification task guide](../tasks/token_classification) on how to use the model.

<PipelineTag pipeline="fill-mask"/>

- [`AlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxAlbertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Masked language modeling task guide](../tasks/masked_language_modeling) on how to use the model.

<PipelineTag pipeline="question-answering"/>

- [`AlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.

**Multiple choice**

- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).


key: _node_content
string_value:
{"id_": "12fb498d-2da7-4e2f-ae6f-6e7f4d763b07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "314949c2-03c8-4ff5-9d64-e296d92c5c6e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "de823b9d7c474be8386dcfc63aff6021bb8ea0ccad4f79ca2f01e4963f692c4d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4300a835-6462-4a76-9d66-fdfd68dc985f", "node_type": "1", "metadata": {}, "hash": "816c4fae17cd9a3519904b5ec9d38fa0aaded8d39b3a74e03e0a39e09cb99646", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6492, "end_char_idx": 8392, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Masked language modeling task guide](../tasks/masked_language_modeling) on how to use the model.

<PipelineTag pipeline="question-answering"/>

- [`AlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.

**Multiple choice**

- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFAlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).

- Check the  [Multiple choice task guide](../tasks/multiple_choice) on how to use the model.


key: _node_content
string_value:
{"id_": "4300a835-6462-4a76-9d66-fdfd68dc985f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d1c30aed-e608-4f55-ac98-da2ab71a54c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f6ba0f36fa613d024afe1881e89cb0d33f8e82842c7d374208948a86d0c2ad11", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12fb498d-2da7-4e2f-ae6f-6e7f4d763b07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/albert/", "title": "ALBERT", "tokens": 2698, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "28376fb110c627aab1112bb713b93ad5b2e1a8377235dce085e6a9f555374967", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7050, "end_char_idx": 10613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxAlbertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- Check the [Question answering task guide](../tasks/question_answering) on how to use the model.

**Multiple choice**

- [`AlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFAlbertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).

- Check the  [Multiple choice task guide](../tasks/multiple_choice) on how to use the model.


## AlbertConfig

[[autodoc]] AlbertConfig

## AlbertTokenizer

[[autodoc]] AlbertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## AlbertTokenizerFast

[[autodoc]] AlbertTokenizerFast

## Albert specific outputs

[[autodoc]] models.albert.modeling_albert.AlbertForPreTrainingOutput

[[autodoc]] models.albert.modeling_tf_albert.TFAlbertForPreTrainingOutput

<frameworkcontent>
<pt>

## AlbertModel

[[autodoc]] AlbertModel
    - forward

## AlbertForPreTraining

[[autodoc]] AlbertForPreTraining
    - forward

## AlbertForMaskedLM

[[autodoc]] AlbertForMaskedLM
    - forward

## AlbertForSequenceClassification

[[autodoc]] AlbertForSequenceClassification
    - forward

## AlbertForMultipleChoice

[[autodoc]] AlbertForMultipleChoice

## AlbertForTokenClassification

[[autodoc]] AlbertForTokenClassification
    - forward

## AlbertForQuestionAnswering

[[autodoc]] AlbertForQuestionAnswering
    - forward

</pt>

<tf>

## TFAlbertModel

[[autodoc]] TFAlbertModel
    - call

## TFAlbertForPreTraining

[[autodoc]] TFAlbertForPreTraining
    - call

## TFAlbertForMaskedLM

[[autodoc]] TFAlbertForMaskedLM
    - call

## TFAlbertForSequenceClassification

[[autodoc]] TFAlbertForSequenceClassification
    - call

## TFAlbertForMultipleChoice

[[autodoc]] TFAlbertForMultipleChoice
    - call

## TFAlbertForTokenClassification

[[autodoc]] TFAlbertForTokenClassification
    - call

## TFAlbertForQuestionAnswering

[[autodoc]] TFAlbertForQuestionAnswering
    - call

</tf>
<jax>

## FlaxAlbertModel

[[autodoc]] FlaxAlbertModel
    - __call__

## FlaxAlbertForPreTraining

[[autodoc]] FlaxAlbertForPreTraining
    - __call__

## FlaxAlbertForMaskedLM

[[autodoc]] FlaxAlbertForMaskedLM
    - __call__

## FlaxAlbertForSequenceClassification

[[autodoc]] FlaxAlbertForSequenceClassification
    - __call__

## FlaxAlbertForMultipleChoice

[[autodoc]] FlaxAlbertForMultipleChoice
    - __call__

## FlaxAlbertForTokenClassification

[[autodoc]] FlaxAlbertForTokenClassification
    - __call__

## FlaxAlbertForQuestionAnswering

[[autodoc]] FlaxAlbertForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "6e537342-8a77-4684-b21e-212832b57fe8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vitdet/", "title": "ViTDet", "tokens": 512, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "813d82b3-adbf-4c9d-9f43-ae7b8cb76271", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vitdet/", "title": "ViTDet", "tokens": 512, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e2b431e2f1f206f1aef277136745dbcdb0ca80cfa165644069936ce0c9862d32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1674, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ViTDet

## Overview

The ViTDet model was proposed in [Exploring Plain Vision Transformer Backbones for Object Detection](https://arxiv.org/abs/2203.16527) by Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He.
VitDet leverages the plain [Vision Transformer](vit) for the task of object detection.

The abstract from the paper is the following:

*We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors.*

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/facebookresearch/detectron2/tree/main/projects/ViTDet).

Tips:

- At the moment, only the backbone is available.

## VitDetConfig

[[autodoc]] VitDetConfig

## VitDetModel

[[autodoc]] VitDetModel
    - forward


key: _node_content
string_value:
{"id_": "abe1fcc1-f79f-4ce1-ad42-20173649c0e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a4b989d63136dcc10e1dfea949c79271c58e33950178b5a4df1b3dd1360cebd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb385bb3-4787-4b4f-9411-ccead91df85d", "node_type": "1", "metadata": {}, "hash": "1aa50d67f9bdc367a17b930ac1561e6255583ee33e52e4d04aae6e0a228b9819", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2582, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Speech2Text

## Overview

The Speech2Text model was proposed in [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://arxiv.org/abs/2010.05171) by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It's a
transformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech
Translation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are
fed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the
transcripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST:
[LibriSpeech](http://www.openslr.org/12), [CoVoST 2](https://github.com/facebookresearch/covost), [MuST-C](https://ict.fbk.eu/must-c/).

This model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text).

## Inference

Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech
signal. It's a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The
`generate()` method can be used for inference.

The [`Speech2TextFeatureExtractor`] class is responsible for extracting the log-mel filter-bank
features. The [`Speech2TextProcessor`] wraps [`Speech2TextFeatureExtractor`] and
[`Speech2TextTokenizer`] into a single instance to both extract the input features and decode the
predicted token ids.

The feature extractor depends on `torchaudio` and the tokenizer depends on `sentencepiece` so be sure to
install those packages before running the examples. You could either install those as extra speech dependencies with
`pip install transformers"[speech, sentencepiece]"` or install the packages separately with `pip install torchaudio sentencepiece`. Also `torchaudio` requires the development version of the [libsndfile](http://www.mega-nerd.com/libsndfile/) package which can be installed via a system package manager. On Ubuntu it can
be installed as follows: `apt install libsndfile1-dev`

- ASR and Speech Translation

```python
>>> import torch
>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
>>> from datasets import load_dataset

>>> model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr")
>>> processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr")


key: _node_content
string_value:
{"id_": "fb385bb3-4787-4b4f-9411-ccead91df85d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fdd5b2d7-8daa-4211-a450-64b6c79d1297", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a4b989d63136dcc10e1dfea949c79271c58e33950178b5a4df1b3dd1360cebd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abe1fcc1-f79f-4ce1-ad42-20173649c0e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text/", "title": "Speech2Text", "tokens": 1503, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a6e95072223f2fea15488be98a29faf7f7e07b4beca0cfd0afd9f101c8846940", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2585, "end_char_idx": 5575, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")

>>> inputs = processor(ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt")
>>> generated_ids = model.generate(inputs["input_features"], attention_mask=inputs["attention_mask"])

>>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> transcription
['mister quilter is the apostle of the middle classes and we are glad to welcome his gospel']
```

- Multilingual speech translation

  For multilingual speech translation models, `eos_token_id` is used as the `decoder_start_token_id` and
  the target language id is forced as the first generated token. To force the target language id as the first
  generated token, pass the `forced_bos_token_id` parameter to the `generate()` method. The following
  example shows how to transate English speech to French text using the *facebook/s2t-medium-mustc-multilingual-st*
  checkpoint.

```python
>>> import torch
>>> from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
>>> from datasets import load_dataset

>>> model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-medium-mustc-multilingual-st")
>>> processor = Speech2TextProcessor.from_pretrained("facebook/s2t-medium-mustc-multilingual-st")

>>> ds = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")

>>> inputs = processor(ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt")
>>> generated_ids = model.generate(
...     inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     forced_bos_token_id=processor.tokenizer.lang_code_to_id["fr"],
... )

>>> translation = processor.batch_decode(generated_ids, skip_special_tokens=True)
>>> translation
["(Vidéo) Si M. Kilder est l'apossible des classes moyennes, et nous sommes heureux d'être accueillis dans son évangile."]
```

See the [model hub](https://huggingface.co/models?filter=speech_to_text) to look for Speech2Text checkpoints.

## Speech2TextConfig

[[autodoc]] Speech2TextConfig

## Speech2TextTokenizer

[[autodoc]] Speech2TextTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## Speech2TextFeatureExtractor

[[autodoc]] Speech2TextFeatureExtractor
    - __call__

## Speech2TextProcessor

[[autodoc]] Speech2TextProcessor
    - __call__
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

<frameworkcontent>
<pt>

## Speech2TextModel

[[autodoc]] Speech2TextModel
    - forward

## Speech2TextForConditionalGeneration

[[autodoc]] Speech2TextForConditionalGeneration
    - forward

</pt>
<tf>

## TFSpeech2TextModel

[[autodoc]] TFSpeech2TextModel
    - call

## TFSpeech2TextForConditionalGeneration

[[autodoc]] TFSpeech2TextForConditionalGeneration
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "c27773de-b80a-4f67-b2b6-508c42b1d2db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/autoformer/", "title": "Autoformer", "tokens": 739, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "da9cf386-6db3-4b70-82b6-2c810c3d6642", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/autoformer/", "title": "Autoformer", "tokens": 739, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4d06474b6b9d7816dd4c9baf1b6120eb5baa8bd401499090073e7e4be39f2505", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2844, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Autoformer

## Overview

The Autoformer model was proposed in [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://arxiv.org/abs/2106.13008) by Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long.

This model augments the Transformer as a deep decomposition architecture, which can progressively decompose the trend and seasonal components during the forecasting process.

The abstract from the paper is the following:

*Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease.*

This model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif).
The original code can be found [here](https://github.com/thuml/Autoformer).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- Check out the Autoformer blog-post in HuggingFace blog: [Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)](https://huggingface.co/blog/autoformer)

## AutoformerConfig

[[autodoc]] AutoformerConfig

## AutoformerModel

[[autodoc]] AutoformerModel
    - forward

## AutoformerForPrediction

[[autodoc]] AutoformerForPrediction
    - forward


