# embedding_metadata

key: _node_content
string_value:
{"id_": "8548434a-2c9c-4527-8c75-4e7c1ced2419", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "hash": "1529d2ae10c4b806c09fc27feb4e8171e68a7d065e61569085d01fab102010ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aacbc404-a05d-4334-90f0-40d7cb99476d", "node_type": "1", "metadata": {}, "hash": "fbdbd3fbf8d1165877f9b1a4377b2ab5653c71138ab696d44704998c1a27a9c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2491, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# KOSMOS-2

## Overview

The KOSMOS-2 model was proposed in [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://arxiv.org/abs/2306.14824) by Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei.

KOSMOS-2 is a Transformer-based causal language model and is trained using the next-word prediction task on a web-scale
dataset of grounded image-text pairs [GRIT](https://huggingface.co/datasets/zzliang/GRIT). The spatial coordinates of
the bounding boxes in the dataset are converted to a sequence of location tokens, which are appended to their respective
entity text spans (for example, `a snowman` followed by `<patch_index_0044><patch_index_0863>`). The data format is
similar to “hyperlinks” that connect the object regions in an image to their text span in the corresponding caption.

The abstract from the paper is the following:

*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg"
alt="drawing" width="600"/>

<small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href="https://arxiv.org/abs/2306.14824">original paper</a>.


key: _node_content
string_value:
{"id_": "aacbc404-a05d-4334-90f0-40d7cb99476d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "hash": "1529d2ae10c4b806c09fc27feb4e8171e68a7d065e61569085d01fab102010ad", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8548434a-2c9c-4527-8c75-4e7c1ced2419", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "hash": "bd54ade80be2dcdfea779330f953e818640345a1e5aa7004881578f588cd3660", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5599913-cba4-43a5-944e-d41315979cdf", "node_type": "1", "metadata": {}, "hash": "989c840e4edc78718d1651302ce8a3570ce373969d02f00bc09ebe110edf467d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 706, "end_char_idx": 3956, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The data format is
similar to “hyperlinks” that connect the object regions in an image to their text span in the corresponding caption.

The abstract from the paper is the following:

*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg"
alt="drawing" width="600"/>

<small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href="https://arxiv.org/abs/2306.14824">original paper</a>. </small>

## Example

```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration

>>> model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")
>>> processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

>>> url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> prompt = "<grounding> An image of"

>>> inputs = processor(text=prompt, images=image, return_tensors="pt")

>>> generated_ids = model.generate(
...     pixel_values=inputs["pixel_values"],
...     input_ids=inputs["input_ids"],
...     attention_mask=inputs["attention_mask"],
...     image_embeds=None,
...     image_embeds_position_mask=inputs["image_embeds_position_mask"],
...     use_cache=True,
...     max_new_tokens=64,
... )
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)
>>> processed_text
'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'

>>> caption, entities = processor.post_process_generation(generated_text)
>>> caption
'An image of a snowman warming himself by a fire.'


key: _node_content
string_value:
{"id_": "f5599913-cba4-43a5-944e-d41315979cdf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "hash": "1529d2ae10c4b806c09fc27feb4e8171e68a7d065e61569085d01fab102010ad", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aacbc404-a05d-4334-90f0-40d7cb99476d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers"}, "hash": "a0151453595caeb110b469eb9588d2d3ed64dd025e64a4a0775398fe9e2630c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2414, "end_char_idx": 4575, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2306.14824">original paper</a>. </small>

## Example

```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration

>>> model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")
>>> processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

>>> url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> prompt = "<grounding> An image of"

>>> inputs = processor(text=prompt, images=image, return_tensors="pt")

>>> generated_ids = model.generate(
...     pixel_values=inputs["pixel_values"],
...     input_ids=inputs["input_ids"],
...     attention_mask=inputs["attention_mask"],
...     image_embeds=None,
...     image_embeds_position_mask=inputs["image_embeds_position_mask"],
...     use_cache=True,
...     max_new_tokens=64,
... )
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)
>>> processed_text
'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'

>>> caption, entities = processor.post_process_generation(generated_text)
>>> caption
'An image of a snowman warming himself by a fire.'

>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```

This model was contributed by [Yih-Dar SHIEH](https://huggingface.co/ydshieh). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/kosmos-2).

## Kosmos2Config

[[autodoc]] Kosmos2Config

## Kosmos2ImageProcessor

## Kosmos2Processor

[[autodoc]] Kosmos2Processor
    - __call__

## Kosmos2Model

[[autodoc]] Kosmos2Model
    - forward

## Kosmos2ForConditionalGeneration

[[autodoc]] Kosmos2ForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "9b35abae-7622-4be5-b767-0a5a610df1a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "af2b61e7-6097-416b-8cfe-26eb68c62d45", "node_type": "1", "metadata": {}, "hash": "a7556100accb5fb4b0ccab26eed5def11201dfcbaccf7da88aa300f1dcd36d9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2939, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Wav2Vec2

## Overview

The Wav2Vec2 model was proposed in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://arxiv.org/abs/2006.11477) by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.

The abstract from the paper is the following:

*We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on
transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks
the speech input in the latent space and solves a contrastive task defined over a quantization of the latent
representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the
clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state
of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and
pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech
recognition with limited amounts of labeled data.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

Note: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).

## Usage tips

- Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].

## Using Flash Attention 2

Flash Attention 2 is an faster, optimized version of the model.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g.


key: _node_content
string_value:
{"id_": "af2b61e7-6097-416b-8cfe-26eb68c62d45", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b35abae-7622-4be5-b767-0a5a610df1a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "ea0a6b070eb0da1d4cb317290c369d891d414269fe09d180334f0f2b6a671c07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b879c1e-31a9-42b3-b893-77724d1f00a9", "node_type": "1", "metadata": {}, "hash": "7db575ae64e35bb846ad4825283746b02ea946b1ccf23258397cebc31ba4f15a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1429, "end_char_idx": 4452, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).

## Usage tips

- Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].

## Using Flash Attention 2

Flash Attention 2 is an faster, optimized version of the model.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
>>> from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("facebook/wav2vec2-large-960h-lv60-self", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
...
```

### Expected speedups

Below is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/wav2vec2-large-960h-lv60-self` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: 


<div style="text-align: center">
<img src="https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Wav2Vec2_speedup.png">
</div>



## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Wav2Vec2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="audio-classification"/>

- A notebook on how to [leverage a pretrained Wav2Vec2 model for emotion classification](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb).


key: _node_content
string_value:
{"id_": "8b879c1e-31a9-42b3-b893-77724d1f00a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "af2b61e7-6097-416b-8cfe-26eb68c62d45", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "a8bb16a2439d0259ae29687d492742fdeb67f2776b4f98f038c15e31333fcad7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8857a361-efae-49b5-ac35-1e453bf7270a", "node_type": "1", "metadata": {}, "hash": "6bcccf13fca0b66112af214c50b34a8ce1dd6fa699a3a6e2143502a6b5dc8435", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2890, "end_char_idx": 5854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
>>> from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("facebook/wav2vec2-large-960h-lv60-self", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
...
```

### Expected speedups

Below is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/wav2vec2-large-960h-lv60-self` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: 


<div style="text-align: center">
<img src="https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Wav2Vec2_speedup.png">
</div>



## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Wav2Vec2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="audio-classification"/>

- A notebook on how to [leverage a pretrained Wav2Vec2 model for emotion classification](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb). 🌎
- [`Wav2Vec2ForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).
- [Audio classification task guide](../tasks/audio_classification)

<PipelineTag pipeline="automatic-speech-recognition"/>

- A blog post on [boosting Wav2Vec2 with n-grams in 🤗 Transformers](https://huggingface.co/blog/wav2vec2-with-ngram).
- A blog post on how to [finetune Wav2Vec2 for English ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english).
- A blog post on [finetuning XLS-R for Multi-Lingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2).
- A notebook on how to [create YouTube captions from any video by transcribing audio with Wav2Vec2](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb). 🌎
- [`Wav2Vec2ForCTC`] is supported by a notebook on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).


key: _node_content
string_value:
{"id_": "8857a361-efae-49b5-ac35-1e453bf7270a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b879c1e-31a9-42b3-b893-77724d1f00a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "6c04bb89c3499e550fd8d86c40e77e9f427a122f56068e4a7c97f8964956ba52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e074af2-c6db-4abe-a93e-bb778292d8f3", "node_type": "1", "metadata": {}, "hash": "cb5249c5093421b669e5163ed416f3eea27e3262955d9020962f1ecd2fee5a64", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4453, "end_char_idx": 7142, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🌎
- [`Wav2Vec2ForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).
- [Audio classification task guide](../tasks/audio_classification)

<PipelineTag pipeline="automatic-speech-recognition"/>

- A blog post on [boosting Wav2Vec2 with n-grams in 🤗 Transformers](https://huggingface.co/blog/wav2vec2-with-ngram).
- A blog post on how to [finetune Wav2Vec2 for English ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english).
- A blog post on [finetuning XLS-R for Multi-Lingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2).
- A notebook on how to [create YouTube captions from any video by transcribing audio with Wav2Vec2](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb). 🌎
- [`Wav2Vec2ForCTC`] is supported by a notebook on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).
- [Automatic speech recognition task guide](../tasks/asr)

🚀 Deploy

- A blog post on how to deploy Wav2Vec2 for [Automatic Speech Recognition with Hugging Face's Transformers & Amazon SageMaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker).

## Wav2Vec2Config

[[autodoc]] Wav2Vec2Config

## Wav2Vec2CTCTokenizer

[[autodoc]] Wav2Vec2CTCTokenizer
    - __call__
    - save_vocabulary
    - decode
    - batch_decode
    - set_target_lang

## Wav2Vec2FeatureExtractor

[[autodoc]] Wav2Vec2FeatureExtractor
    - __call__

## Wav2Vec2Processor

[[autodoc]] Wav2Vec2Processor
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## Wav2Vec2ProcessorWithLM

[[autodoc]] Wav2Vec2ProcessorWithLM
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

### Decoding multiple audios

If you are planning to decode multiple batches of audios, you should consider using [`~Wav2Vec2ProcessorWithLM.batch_decode`] and passing an instantiated `multiprocessing.Pool`.
Otherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call.


key: _node_content
string_value:
{"id_": "5e074af2-c6db-4abe-a93e-bb778292d8f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8857a361-efae-49b5-ac35-1e453bf7270a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "04679e25c1d70a71e739a8cb72a5f728edaae934988912c5e0a1a4568e41cb0d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b0f510a-0ed3-49e0-b70f-490ed5431744", "node_type": "1", "metadata": {}, "hash": "7cb29b1fc564efa70e0b6542d82929556ce1233fc00a29d8734cc2e556116205", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5855, "end_char_idx": 8546, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Automatic speech recognition task guide](../tasks/asr)

🚀 Deploy

- A blog post on how to deploy Wav2Vec2 for [Automatic Speech Recognition with Hugging Face's Transformers & Amazon SageMaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker).

## Wav2Vec2Config

[[autodoc]] Wav2Vec2Config

## Wav2Vec2CTCTokenizer

[[autodoc]] Wav2Vec2CTCTokenizer
    - __call__
    - save_vocabulary
    - decode
    - batch_decode
    - set_target_lang

## Wav2Vec2FeatureExtractor

[[autodoc]] Wav2Vec2FeatureExtractor
    - __call__

## Wav2Vec2Processor

[[autodoc]] Wav2Vec2Processor
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## Wav2Vec2ProcessorWithLM

[[autodoc]] Wav2Vec2ProcessorWithLM
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

### Decoding multiple audios

If you are planning to decode multiple batches of audios, you should consider using [`~Wav2Vec2ProcessorWithLM.batch_decode`] and passing an instantiated `multiprocessing.Pool`.
Otherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call. See the example below:

```python
>>> # Let's see how to use a user-managed pool for batch decoding multiple audios
>>> from multiprocessing import get_context
>>> from transformers import AutoTokenizer, AutoProcessor, AutoModelForCTC
>>> from datasets import load_dataset
>>> import datasets
>>> import torch

>>> # import model, feature extractor, tokenizer
>>> model = AutoModelForCTC.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm").to("cuda")
>>> processor = AutoProcessor.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")

>>> # load example dataset
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))


>>> def map_to_array(batch):
...     batch["speech"] = batch["audio"]["array"]
...     return batch


>>> # prepare speech data for batch inference
>>> dataset = dataset.map(map_to_array, remove_columns=["audio"])


>>> def map_to_pred(batch, pool):
...     inputs = processor(batch["speech"], sampling_rate=16_000, padding=True, return_tensors="pt")
...     inputs = {k: v.to("cuda") for k, v in inputs.items()}

...     with torch.no_grad():
...         logits = model(**inputs).logits

...     transcription = processor.batch_decode(logits.cpu().numpy(), pool).text
...     batch["transcription"] = transcription
...     return batch


key: _node_content
string_value:
{"id_": "0b0f510a-0ed3-49e0-b70f-490ed5431744", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e074af2-c6db-4abe-a93e-bb778292d8f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers"}, "hash": "e3db3713dba1c9dabbc5e86495602ff978b3b962b6a6ef290d268fd852ee6256", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6926, "end_char_idx": 10652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Otherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call. See the example below:

```python
>>> # Let's see how to use a user-managed pool for batch decoding multiple audios
>>> from multiprocessing import get_context
>>> from transformers import AutoTokenizer, AutoProcessor, AutoModelForCTC
>>> from datasets import load_dataset
>>> import datasets
>>> import torch

>>> # import model, feature extractor, tokenizer
>>> model = AutoModelForCTC.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm").to("cuda")
>>> processor = AutoProcessor.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")

>>> # load example dataset
>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000))


>>> def map_to_array(batch):
...     batch["speech"] = batch["audio"]["array"]
...     return batch


>>> # prepare speech data for batch inference
>>> dataset = dataset.map(map_to_array, remove_columns=["audio"])


>>> def map_to_pred(batch, pool):
...     inputs = processor(batch["speech"], sampling_rate=16_000, padding=True, return_tensors="pt")
...     inputs = {k: v.to("cuda") for k, v in inputs.items()}

...     with torch.no_grad():
...         logits = model(**inputs).logits

...     transcription = processor.batch_decode(logits.cpu().numpy(), pool).text
...     batch["transcription"] = transcription
...     return batch


>>> # note: pool should be instantiated *after* `Wav2Vec2ProcessorWithLM`.
>>> #       otherwise, the LM won't be available to the pool's sub-processes
>>> # select number of processes and batch_size based on number of CPU cores available and on dataset size
>>> with get_context("fork").Pool(processes=2) as pool:
...     result = dataset.map(
...         map_to_pred, batched=True, batch_size=2, fn_kwargs={"pool": pool}, remove_columns=["speech"]
...     )

>>> result["transcription"][:2]
['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL', "NOR IS MISTER COULTER'S MANNER LESS INTERESTING THAN HIS MATTER"]
```

## Wav2Vec2 specific outputs

[[autodoc]] models.wav2vec2_with_lm.processing_wav2vec2_with_lm.Wav2Vec2DecoderWithLMOutput

[[autodoc]] models.wav2vec2.modeling_wav2vec2.Wav2Vec2BaseModelOutput

[[autodoc]] models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTrainingOutput

[[autodoc]] models.wav2vec2.modeling_flax_wav2vec2.FlaxWav2Vec2BaseModelOutput

[[autodoc]] models.wav2vec2.modeling_flax_wav2vec2.FlaxWav2Vec2ForPreTrainingOutput

<frameworkcontent>
<pt>

## Wav2Vec2Model

[[autodoc]] Wav2Vec2Model
    - forward

## Wav2Vec2ForCTC

[[autodoc]] Wav2Vec2ForCTC
    - forward
    - load_adapter

## Wav2Vec2ForSequenceClassification

[[autodoc]] Wav2Vec2ForSequenceClassification
    - forward

## Wav2Vec2ForAudioFrameClassification

[[autodoc]] Wav2Vec2ForAudioFrameClassification
    - forward

## Wav2Vec2ForXVector

[[autodoc]] Wav2Vec2ForXVector
    - forward

## Wav2Vec2ForPreTraining

[[autodoc]] Wav2Vec2ForPreTraining
    - forward

</pt>
<tf>

## TFWav2Vec2Model

[[autodoc]] TFWav2Vec2Model
    - call

## TFWav2Vec2ForSequenceClassification

[[autodoc]] TFWav2Vec2ForSequenceClassification
    - call

## TFWav2Vec2ForCTC

[[autodoc]] TFWav2Vec2ForCTC
    - call

</tf>
<jax>

## FlaxWav2Vec2Model

[[autodoc]] FlaxWav2Vec2Model
    - __call__

## FlaxWav2Vec2ForCTC

[[autodoc]] FlaxWav2Vec2ForCTC
    - __call__

## FlaxWav2Vec2ForPreTraining

[[autodoc]] FlaxWav2Vec2ForPreTraining
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "41618e4f-b8b7-4e03-ab10-20d4c47de2fe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt-sw3", "title": "GPT-Sw3", "tokens": 727, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae3a9ea3-7ae7-4c0a-9a1e-542a9db90955", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt-sw3", "title": "GPT-Sw3", "tokens": 727, "retrieve_doc": true, "source": "transformers"}, "hash": "6750a43e5a76ee025437d6599bb92f92bfe77d5345e000ddd08dfa29b628f405", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2216, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPT-Sw3

## Overview

The GPT-Sw3 model was first proposed in
[Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish](http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.376.pdf)
by Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman,
Fredrik Carlsson, Magnus Sahlgren.

Since that first paper the authors have extended their work and trained new models on their new 1.2TB corpora named The Nordic Pile.

GPT-Sw3 is a collection of large decoder-only pretrained transformer language models that were developed by AI Sweden
in collaboration with RISE and the WASP WARA for Media and Language. GPT-Sw3 has been trained on a dataset containing
320B tokens in Swedish, Norwegian, Danish, Icelandic, English, and programming code. The model was pretrained using a
causal language modeling (CLM) objective utilizing the NeMo Megatron GPT implementation.

This model was contributed by [AI Sweden Models](https://huggingface.co/AI-Sweden-Models).

## Usage example

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("AI-Sweden-Models/gpt-sw3-356m")
>>> model = AutoModelForCausalLM.from_pretrained("AI-Sweden-Models/gpt-sw3-356m")

>>> input_ids = tokenizer("Träd är fina för att", return_tensors="pt")["input_ids"]

>>> generated_token_ids = model.generate(inputs=input_ids, max_new_tokens=10, do_sample=True)[0]

>>> print(tokenizer.decode(generated_token_ids))
Träd är fina för att de är färgstarka. Men ibland är det fint
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

<Tip>

The implementation uses the `GPT2Model` coupled with our `GPTSw3Tokenizer`. Refer to [GPT2Model documentation](gpt2) 
for API reference and examples.  

Note that sentencepiece is required to use our tokenizer and can be installed with `pip install transformers[sentencepiece]` or `pip install sentencepiece`

</Tip>

## GPTSw3Tokenizer

[[autodoc]] GPTSw3Tokenizer
    - save_vocabulary


key: _node_content
string_value:
{"id_": "63dd9c04-699f-4d01-8ae2-b720fb574244", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vivit", "title": "Video Vision Transformer (ViViT)", "tokens": 546, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f1ac533-5083-4fd7-bbce-7b0af2756f3e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vivit", "title": "Video Vision Transformer (ViViT)", "tokens": 546, "retrieve_doc": true, "source": "transformers"}, "hash": "81b76087ebefb51cf04926f819331492e567ab75140077acea2eeace33b70520", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1878, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Video Vision Transformer (ViViT)

## Overview

The Vivit model was proposed in [ViViT: A Video Vision Transformer](https://arxiv.org/abs/2103.15691) by Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid.
The paper proposes one of the first successful pure-transformer based set of models for video understanding.

The abstract from the paper is the following:

*We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks.*

This model was contributed by [jegormeister](https://huggingface.co/jegormeister). The original code (written in JAX) can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/vivit).

## VivitConfig

[[autodoc]] VivitConfig

## VivitImageProcessor

[[autodoc]] VivitImageProcessor
    - preprocess

## VivitModel

[[autodoc]] VivitModel
    - forward

## VivitForVideoClassification

[[autodoc]] transformers.VivitForVideoClassification
    - forward


key: _node_content
string_value:
{"id_": "f9533a70-56ef-4c07-bd7f-bc2ceab60b90", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers"}, "hash": "a180558ea7ddc93a3fd1d4c593f79ba78831de05cd2f2a4ef5312691c6ff1545", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3732c6a2-e154-4f85-929b-1785aa596010", "node_type": "1", "metadata": {}, "hash": "1cb00cc3a9758088f1fa4dca3c0879a6333099dad9a822387e59dde449ef047c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3320, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ResNet

## Overview

The ResNet model was proposed in [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Our implementation follows the small changes made by [Nvidia](https://catalog.ngc.nvidia.com/orgs/nvidia/resources/resnet_50_v1_5_for_pytorch), we apply the `stride=2` for downsampling in bottleneck's `3x3` conv and not in the first `1x1`. This is generally known as "ResNet v1.5".

ResNet introduced residual connections, they allow to train networks with an unseen number of layers (up to 1000). ResNet won the 2015 ILSVRC & COCO competition, one important milestone in deep computer vision.

The abstract from the paper is the following:

*Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.*

The figure below illustrates the architecture of ResNet. Taken from the [original paper](https://arxiv.org/abs/1512.03385).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/resnet_architecture.png"/>

This model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/KaimingHe/deep-residual-networks).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ResNet.

<PipelineTag pipeline="image-classification"/>

- [`ResNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "3732c6a2-e154-4f85-929b-1785aa596010", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers"}, "hash": "a180558ea7ddc93a3fd1d4c593f79ba78831de05cd2f2a4ef5312691c6ff1545", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f9533a70-56ef-4c07-bd7f-bc2ceab60b90", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers"}, "hash": "b6dd7c59ca940db96ecfa36eccea84122226414e047e8ad195414c0564e2344e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1590, "end_char_idx": 3893, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.*

The figure below illustrates the architecture of ResNet. Taken from the [original paper](https://arxiv.org/abs/1512.03385).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/resnet_architecture.png"/>

This model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/KaimingHe/deep-residual-networks).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ResNet.

<PipelineTag pipeline="image-classification"/>

- [`ResNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ResNetConfig

[[autodoc]] ResNetConfig

<frameworkcontent>
<pt>

## ResNetModel

[[autodoc]] ResNetModel
    - forward

## ResNetForImageClassification

[[autodoc]] ResNetForImageClassification
    - forward

</pt>
<tf>

## TFResNetModel

[[autodoc]] TFResNetModel
    - call

## TFResNetForImageClassification

[[autodoc]] TFResNetForImageClassification
    - call

</tf>
<jax>

## FlaxResNetModel

[[autodoc]] FlaxResNetModel
    - __call__

## FlaxResNetForImageClassification

[[autodoc]] FlaxResNetForImageClassification
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "627f06b8-dee4-4c13-afb4-08cde39aeea6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/van", "title": "VAN", "tokens": 910, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "380a5858-5faf-4e03-b684-dba2d9ea731c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/van", "title": "VAN", "tokens": 910, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f375d38bd35bfae59a1ee8570cb707c4b64710d325f5700eac19745f0f390e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3445, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VAN

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.

If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.
You can do so by running the following command: `pip install -U transformers==4.30.0`.

</Tip>

## Overview

The VAN model was proposed in [Visual Attention Network](https://arxiv.org/abs/2202.09741) by Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu.

This paper introduces a new attention layer based on convolution operations able to capture both local and distant relationships. This is done by combining normal and large kernel convolution layers. The latter uses a dilated convolution to capture distant correlations.

The abstract from the paper is the following:

*While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel large kernel attention (LKA) module to enable self-adaptive and long-range correlations in self-attention while avoiding the above issues. We further introduce a novel neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN outperforms the state-of-the-art vision transformers and convolutional neural networks with a large margin in extensive experiments, including image classification, object detection, semantic segmentation, instance segmentation, etc. Code is available at [this https URL](https://github.com/Visual-Attention-Network/VAN-Classification).*

Tips:

- VAN does not have an embedding layer, thus the `hidden_states` will have a length equal to the number of stages.

The figure below illustrates the architecture of a Visual Attention Layer. Taken from the [original paper](https://arxiv.org/abs/2202.09741).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/van_architecture.png"/>

This model was contributed by [Francesco](https://huggingface.co/Francesco). The original code can be found [here](https://github.com/Visual-Attention-Network/VAN-Classification).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with VAN.

<PipelineTag pipeline="image-classification"/>

- [`VanForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## VanConfig

[[autodoc]] VanConfig

## VanModel

[[autodoc]] VanModel
    - forward

## VanForImageClassification

[[autodoc]] VanForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "12936783-e88f-4b63-941d-14b93d5c7676", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers"}, "hash": "b4cb55347c115a5eb94694b83b507f7ba9c4670f671d4f48760b09ecee768c4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eeef13fb-c29a-4e86-90e6-1d32efe44fda", "node_type": "1", "metadata": {}, "hash": "8af9a48db5e31ca8071c07a3de21b1a02f51533548a9c6c829c8fb3dce797b10", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2269, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FlauBERT

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=flaubert">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-flaubert-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/flaubert_small_cased">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The FlauBERT model was proposed in the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le et al. It's a transformer model pretrained using a masked language
modeling (MLM) objective (like BERT).

The abstract from the paper is the following:

*Language models have become a key step to achieve state-of-the art results in many different Natural Language
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way
to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
contextualization at the sentence level. This has been widely demonstrated for English using contextualized
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,
2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and
heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for
Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the
time they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation
protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research
community for further reproducible experiments in French NLP.*

This model was contributed by [formiel](https://huggingface.co/formiel). The original code can be found [here](https://github.com/getalp/Flaubert).

Tips:
- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective).


key: _node_content
string_value:
{"id_": "eeef13fb-c29a-4e86-90e6-1d32efe44fda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers"}, "hash": "b4cb55347c115a5eb94694b83b507f7ba9c4670f671d4f48760b09ecee768c4a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12936783-e88f-4b63-941d-14b93d5c7676", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers"}, "hash": "a04ac287db6a646c07704d752bf61fcf734e7a330b4111ae8ad9d8b57ff483a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 559, "end_char_idx": 3902, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It's a transformer model pretrained using a masked language
modeling (MLM) objective (like BERT).

The abstract from the paper is the following:

*Language models have become a key step to achieve state-of-the art results in many different Natural Language
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way
to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
contextualization at the sentence level. This has been widely demonstrated for English using contextualized
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,
2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and
heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for
Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the
time they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation
protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research
community for further reproducible experiments in French NLP.*

This model was contributed by [formiel](https://huggingface.co/formiel). The original code can be found [here](https://github.com/getalp/Flaubert).

Tips:
- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## FlaubertConfig

[[autodoc]] FlaubertConfig

## FlaubertTokenizer

[[autodoc]] FlaubertTokenizer

<frameworkcontent>
<pt>

## FlaubertModel

[[autodoc]] FlaubertModel
    - forward

## FlaubertWithLMHeadModel

[[autodoc]] FlaubertWithLMHeadModel
    - forward

## FlaubertForSequenceClassification

[[autodoc]] FlaubertForSequenceClassification
    - forward

## FlaubertForMultipleChoice

[[autodoc]] FlaubertForMultipleChoice
    - forward

## FlaubertForTokenClassification

[[autodoc]] FlaubertForTokenClassification
    - forward

## FlaubertForQuestionAnsweringSimple

[[autodoc]] FlaubertForQuestionAnsweringSimple
    - forward

## FlaubertForQuestionAnswering

[[autodoc]] FlaubertForQuestionAnswering
    - forward

</pt>
<tf>

## TFFlaubertModel

[[autodoc]] TFFlaubertModel
    - call

## TFFlaubertWithLMHeadModel

[[autodoc]] TFFlaubertWithLMHeadModel
    - call

## TFFlaubertForSequenceClassification

[[autodoc]] TFFlaubertForSequenceClassification
    - call

## TFFlaubertForMultipleChoice

[[autodoc]] TFFlaubertForMultipleChoice
    - call

## TFFlaubertForTokenClassification

[[autodoc]] TFFlaubertForTokenClassification
    - call

## TFFlaubertForQuestionAnsweringSimple

[[autodoc]] TFFlaubertForQuestionAnsweringSimple
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8bba8785-0388-4331-96d8-db62d1a7cc1a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c678f14e-6613-40fa-8482-e44865016eec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29a14a8a-fc0c-400d-a808-70575ed51b61", "node_type": "1", "metadata": {}, "hash": "52c6c2d35a21e77b21af6968aed82910cc28c1ea417ea194e450e40142b01820", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3338, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TVP

## Overview

The text-visual prompting (TVP) framework was proposed in the paper [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://arxiv.org/abs/2303.04995) by Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding.

The abstract from the paper is the following:

*In this paper, we study the problem of temporal video grounding (TVG), which aims to predict the starting/ending time points of moments described by a text sentence within a long untrimmed video. Benefiting from fine-grained 3D visual features, the TVG techniques have achieved remarkable progress in recent years. However, the high complexity of 3D convolutional neural networks (CNNs) makes extracting dense 3D visual features time-consuming, which calls for intensive memory and computing resources. Towards efficient TVG, we propose a novel text-visual prompting (TVP) framework, which incorporates optimized perturbation patterns (that we call ‘prompts’) into both visual inputs and textual features of a TVG model. In sharp contrast to 3D CNNs, we show that TVP allows us to effectively co-train vision encoder and language encoder in a 2D TVG model and improves the performance of cross-modal feature fusion using only low-complexity sparse 2D visual features. Further, we propose a Temporal-Distance IoU (TDIoU) loss for efficient learning of TVG. Experiments on two benchmark datasets, Charades-STA and ActivityNet Captions datasets, empirically show that the proposed TVP significantly boosts the performance of 2D TVG (e.g., 9.79% improvement on Charades-STA and 30.77% improvement on ActivityNet Captions) and achieves 5× inference acceleration over TVG using 3D visual features.*

This research addresses temporal video grounding (TVG), which is the process of pinpointing the start and end times of specific events in a long video, as described by a text sentence. Text-visual prompting (TVP), is proposed to enhance TVG. TVP involves integrating specially designed patterns, known as 'prompts', into both the visual (image-based) and textual (word-based) input components of a TVG model. These prompts provide additional spatial-temporal context, improving the model's ability to accurately determine event timings in the video. The approach employs 2D visual inputs in place of 3D ones. Although 3D inputs offer more spatial-temporal detail, they are also more time-consuming to process. The use of 2D inputs with the prompting method aims to provide similar levels of context and accuracy more efficiently.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png"
alt="drawing" width="600"/>

<small> TVP architecture. Taken from the <a href="https://arxiv.org/abs/2303.04995">original paper.</a> </small>

This model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).

## Usage tips and examples

Prompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.

TVP consists of a visual encoder and cross-modal encoder.


key: _node_content
string_value:
{"id_": "29a14a8a-fc0c-400d-a808-70575ed51b61", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c678f14e-6613-40fa-8482-e44865016eec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8bba8785-0388-4331-96d8-db62d1a7cc1a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "8e4d4ef49ef34869cf8191a35ed10b85ea8770b505cecd42c4e3a2e8034ac5a3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50a66a9c-791e-419f-a512-ba5394880b57", "node_type": "1", "metadata": {}, "hash": "04e76286dc9cbeb8bc68dc4421ef5f180059f9b77b56d0abc68d68b3523c4af5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1688, "end_char_idx": 4266, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

This research addresses temporal video grounding (TVG), which is the process of pinpointing the start and end times of specific events in a long video, as described by a text sentence. Text-visual prompting (TVP), is proposed to enhance TVG. TVP involves integrating specially designed patterns, known as 'prompts', into both the visual (image-based) and textual (word-based) input components of a TVG model. These prompts provide additional spatial-temporal context, improving the model's ability to accurately determine event timings in the video. The approach employs 2D visual inputs in place of 3D ones. Although 3D inputs offer more spatial-temporal detail, they are also more time-consuming to process. The use of 2D inputs with the prompting method aims to provide similar levels of context and accuracy more efficiently.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png"
alt="drawing" width="600"/>

<small> TVP architecture. Taken from the <a href="https://arxiv.org/abs/2303.04995">original paper.</a> </small>

This model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).

## Usage tips and examples

Prompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.

TVP consists of a visual encoder and cross-modal encoder. A universal set of visual prompts and text prompts to be integrated into sampled video frames and textual features, respectively. Specially, a set of different visual prompts are applied to uniformly-sampled frames of one untrimmed video in order.

The goal of this model is to incorporate trainable prompts into both visual inputs and textual features to temporal video grounding(TVG) problems.
In principle, one can apply any visual, cross-modal encoder in the proposed architecture.

The [`TvpProcessor`] wraps [`BertTokenizer`] and [`TvpImageProcessor`] into a single instance to both
encode the text and prepare the images respectively.

The following example shows how to run temporal video grounding using [`TvpProcessor`] and [`TvpForVideoGrounding`].
```python
import av
import cv2
import numpy as np
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor, TvpForVideoGrounding


key: _node_content
string_value:
{"id_": "50a66a9c-791e-419f-a512-ba5394880b57", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c678f14e-6613-40fa-8482-e44865016eec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29a14a8a-fc0c-400d-a808-70575ed51b61", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "f71fe28eba73cd48724aaca12889dc3b59d11ae5030cf7f6aaa0b5d15ea05aae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eb9fe06c-cb75-4d07-962f-02b58b47d309", "node_type": "1", "metadata": {}, "hash": "6f83deaeaed929a8cc079e479a2187704f78b0faa245c7a72eef01c44edc7a2e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2522, "end_char_idx": 6125, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png"
alt="drawing" width="600"/>

<small> TVP architecture. Taken from the <a href="https://arxiv.org/abs/2303.04995">original paper.</a> </small>

This model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).

## Usage tips and examples

Prompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.

TVP consists of a visual encoder and cross-modal encoder. A universal set of visual prompts and text prompts to be integrated into sampled video frames and textual features, respectively. Specially, a set of different visual prompts are applied to uniformly-sampled frames of one untrimmed video in order.

The goal of this model is to incorporate trainable prompts into both visual inputs and textual features to temporal video grounding(TVG) problems.
In principle, one can apply any visual, cross-modal encoder in the proposed architecture.

The [`TvpProcessor`] wraps [`BertTokenizer`] and [`TvpImageProcessor`] into a single instance to both
encode the text and prepare the images respectively.

The following example shows how to run temporal video grounding using [`TvpProcessor`] and [`TvpForVideoGrounding`].
```python
import av
import cv2
import numpy as np
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor, TvpForVideoGrounding


def pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps):
    '''
    Convert the video from its original fps to the target_fps and decode the video with PyAV decoder.
    Args:
        container (container): pyav container.
        sampling_rate (int): frame sampling rate (interval between two sampled frames).
        num_frames (int): number of frames to sample.
        clip_idx (int): if clip_idx is -1, perform random temporal sampling.
            If clip_idx is larger than -1, uniformly split the video to num_clips
            clips, and select the clip_idx-th video clip.
        num_clips (int): overall number of clips to uniformly sample from the given video.
        target_fps (int): the input video may have different fps, convert it to
            the target video fps before frame sampling.
    Returns:
        frames (tensor): decoded frames from the video. Return None if the no
            video stream was found.
        fps (float): the number of frames per second of the video.
    '''
    video = container.streams.video[0]
    fps = float(video.average_rate)
    clip_size = sampling_rate * num_frames / target_fps * fps
    delta = max(num_frames - clip_size, 0)
    start_idx = delta * clip_idx / num_clips
    end_idx = start_idx + clip_size - 1
    timebase = video.duration / num_frames
    video_start_pts = int(start_idx * timebase)
    video_end_pts = int(end_idx * timebase)
    seek_offset = max(video_start_pts - 1024, 0)
    container.seek(seek_offset, any_frame=False, backward=True, stream=video)
    frames = {}
    for frame in container.decode(video=0):
        if frame.pts < video_start_pts:
            continue
        frames[frame.pts] = frame
        if frame.pts > video_end_pts:
            break
    frames = [frames[pts] for pts in sorted(frames)]
    return frames, fps


key: _node_content
string_value:
{"id_": "eb9fe06c-cb75-4d07-962f-02b58b47d309", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c678f14e-6613-40fa-8482-e44865016eec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50a66a9c-791e-419f-a512-ba5394880b57", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "cdf9c09ed2b4900b3dceb74705ab2b2597387d62ed85fed324c2b39bda96514b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d844e03-13bc-4660-90e9-be0a7846186e", "node_type": "1", "metadata": {}, "hash": "1d008542f6e1250f373683d4b89bb1e9a5cc15926d23a73c36c8aa343b0a84eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6128, "end_char_idx": 9006, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps):
    '''
    Decode the video and perform temporal sampling.
    Args:
        container (container): pyav container.
        sampling_rate (int): frame sampling rate (interval between two sampled frames).
        num_frames (int): number of frames to sample.
        clip_idx (int): if clip_idx is -1, perform random temporal sampling.
            If clip_idx is larger than -1, uniformly split the video to num_clips
            clips, and select the clip_idx-th video clip.
        num_clips (int): overall number of clips to uniformly sample from the given video.
        target_fps (int): the input video may have different fps, convert it to
            the target video fps before frame sampling.
    Returns:
        frames (tensor): decoded frames from the video.
    '''
    assert clip_idx >= -2, "Not a valied clip_idx {}".format(clip_idx)
    frames, fps = pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps)
    clip_size = sampling_rate * num_frames / target_fps * fps
    index = np.linspace(0, clip_size - 1, num_frames)
    index = np.clip(index, 0, len(frames) - 1).astype(np.int64)
    frames = np.array([frames[idx].to_rgb().to_ndarray() for idx in index])
    frames = frames.transpose(0, 3, 1, 2)
    return frames


file = hf_hub_download(repo_id="Intel/tvp_demo", filename="AK2KG.mp4", repo_type="dataset")
model = TvpForVideoGrounding.from_pretrained("Intel/tvp-base")

decoder_kwargs = dict(
    container=av.open(file, metadata_errors="ignore"),
    sampling_rate=1,
    num_frames=model.config.num_frames,
    clip_idx=0,
    num_clips=1,
    target_fps=3,
)
raw_sampled_frms = decode(**decoder_kwargs)

text = "a person is sitting on a bed."
processor = AutoProcessor.from_pretrained("Intel/tvp-base")
model_inputs = processor(
    text=[text], videos=list(raw_sampled_frms), return_tensors="pt", max_text_length=100#, size=size
)

model_inputs["pixel_values"] = model_inputs["pixel_values"].to(model.dtype)
output = model(**model_inputs)

def get_video_duration(filename):
    cap = cv2.VideoCapture(filename)
    if cap.isOpened():
        rate = cap.get(5)
        frame_num = cap.get(7)
        duration = frame_num/rate
        return duration
    return -1

duration = get_video_duration(file)
start, end = processor.post_process_video_grounding(output.logits, duration)

print(f"The time slot of the video corresponding to the text \"{text}\" is from {start}s to {end}s")
```

Tips:

- This implementation of TVP uses [`BertTokenizer`] to generate text embeddings and Resnet-50 model to compute visual embeddings.
- Checkpoints for pre-trained [tvp-base](https://huggingface.co/Intel/tvp-base) is released.
- Please refer to [Table 2](https://arxiv.org/pdf/2303.04995.pdf) for TVP's performance on Temporal Video Grounding task.


key: _node_content
string_value:
{"id_": "8d844e03-13bc-4660-90e9-be0a7846186e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c678f14e-6613-40fa-8482-e44865016eec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eb9fe06c-cb75-4d07-962f-02b58b47d309", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers"}, "hash": "52789233a75c04d717bcdc3f42ee8c9050b7ae03ae6321de48efd575b2f86bde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7481, "end_char_idx": 9294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
file = hf_hub_download(repo_id="Intel/tvp_demo", filename="AK2KG.mp4", repo_type="dataset")
model = TvpForVideoGrounding.from_pretrained("Intel/tvp-base")

decoder_kwargs = dict(
    container=av.open(file, metadata_errors="ignore"),
    sampling_rate=1,
    num_frames=model.config.num_frames,
    clip_idx=0,
    num_clips=1,
    target_fps=3,
)
raw_sampled_frms = decode(**decoder_kwargs)

text = "a person is sitting on a bed."
processor = AutoProcessor.from_pretrained("Intel/tvp-base")
model_inputs = processor(
    text=[text], videos=list(raw_sampled_frms), return_tensors="pt", max_text_length=100#, size=size
)

model_inputs["pixel_values"] = model_inputs["pixel_values"].to(model.dtype)
output = model(**model_inputs)

def get_video_duration(filename):
    cap = cv2.VideoCapture(filename)
    if cap.isOpened():
        rate = cap.get(5)
        frame_num = cap.get(7)
        duration = frame_num/rate
        return duration
    return -1

duration = get_video_duration(file)
start, end = processor.post_process_video_grounding(output.logits, duration)

print(f"The time slot of the video corresponding to the text \"{text}\" is from {start}s to {end}s")
```

Tips:

- This implementation of TVP uses [`BertTokenizer`] to generate text embeddings and Resnet-50 model to compute visual embeddings.
- Checkpoints for pre-trained [tvp-base](https://huggingface.co/Intel/tvp-base) is released.
- Please refer to [Table 2](https://arxiv.org/pdf/2303.04995.pdf) for TVP's performance on Temporal Video Grounding task.


## TvpConfig

[[autodoc]] TvpConfig

## TvpImageProcessor

[[autodoc]] TvpImageProcessor
    - preprocess

## TvpProcessor

[[autodoc]] TvpProcessor
    - __call__

## TvpModel

[[autodoc]] TvpModel
    - forward

## TvpForVideoGrounding

[[autodoc]] TvpForVideoGrounding
    - forward


key: _node_content
string_value:
{"id_": "a50615ca-492a-40db-b858-f95b9814e1d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers"}, "hash": "d0f3d6e2fb79f7d523961ad90a4348c767b483480f117ea156a0fcb3cde7a26c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12d0183b-6923-4d19-8e34-46a0c1dbfe9b", "node_type": "1", "metadata": {}, "hash": "efdf391577bc1f2dc0c44dbc8bf420acfd67b596e35ddc5964401d114c481485", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FNet

## Overview

The FNet model was proposed in [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. The model replaces the self-attention layer in a BERT
model with a fourier transform which returns only the real parts of the transform. The model is significantly faster
than the BERT model because it has fewer parameters and is more memory efficient. The model achieves about 92-97%
accuracy of BERT counterparts on GLUE benchmark, and trains much faster than the BERT model. The abstract from the
paper is the following:

*We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the
self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with
standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text
classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder
with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE
benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths,
our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena
benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all
sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint
and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models
outperform Transformer counterparts.*

This model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/google-research/google-research/tree/master/f_net).

## Usage tips

The model was trained without an attention mask as it is based on Fourier Transform. The model was trained with 
maximum sequence length 512 which includes pad tokens. Hence, it is highly recommended to use the same maximum 
sequence length for fine-tuning and inference.


key: _node_content
string_value:
{"id_": "12d0183b-6923-4d19-8e34-46a0c1dbfe9b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers"}, "hash": "d0f3d6e2fb79f7d523961ad90a4348c767b483480f117ea156a0fcb3cde7a26c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a50615ca-492a-40db-b858-f95b9814e1d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers"}, "hash": "65eb9c05e416ac614e89377c6d3c9277561ac24913731ae12cb5c3b301e012d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 454, "end_char_idx": 3508, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The model achieves about 92-97%
accuracy of BERT counterparts on GLUE benchmark, and trains much faster than the BERT model. The abstract from the
paper is the following:

*We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the
self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with
standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text
classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder
with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE
benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths,
our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena
benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all
sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint
and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models
outperform Transformer counterparts.*

This model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/google-research/google-research/tree/master/f_net).

## Usage tips

The model was trained without an attention mask as it is based on Fourier Transform. The model was trained with 
maximum sequence length 512 which includes pad tokens. Hence, it is highly recommended to use the same maximum 
sequence length for fine-tuning and inference.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## FNetConfig

[[autodoc]] FNetConfig

## FNetTokenizer

[[autodoc]] FNetTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## FNetTokenizerFast

[[autodoc]] FNetTokenizerFast

## FNetModel

[[autodoc]] FNetModel
    - forward

## FNetForPreTraining

[[autodoc]] FNetForPreTraining
    - forward

## FNetForMaskedLM

[[autodoc]] FNetForMaskedLM
    - forward

## FNetForNextSentencePrediction

[[autodoc]] FNetForNextSentencePrediction
    - forward

## FNetForSequenceClassification

[[autodoc]] FNetForSequenceClassification
    - forward

## FNetForMultipleChoice

[[autodoc]] FNetForMultipleChoice
    - forward

## FNetForTokenClassification

[[autodoc]] FNetForTokenClassification
    - forward

## FNetForQuestionAnswering

[[autodoc]] FNetForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "f74514fa-00ff-4bcd-a6e8-5f25606186f2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/unispeech", "title": "UniSpeech", "tokens": 731, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a08dae22-cee2-47ec-89a7-1c7ffa87e8f4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/unispeech", "title": "UniSpeech", "tokens": 731, "retrieve_doc": true, "source": "transformers"}, "hash": "8577f7a3225609506b83423b7ed57c01159310ecfba559af75d9402fb98b494e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# UniSpeech

## Overview

The UniSpeech model was proposed in [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://arxiv.org/abs/2101.07597) by Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael
Zeng, Xuedong Huang .

The abstract from the paper is the following:

*In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both
unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive
self-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture
information more correlated with phonetic structures and improve the generalization across languages and domains. We
evaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The
results show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech
recognition by a maximum of 13.4% and 17.8% relative phone error rate reductions respectively (averaged over all
testing languages). The transferability of UniSpeech is also demonstrated on a domain-shift speech recognition task,
i.e., a relative word error rate reduction of 6% against the previous approach.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be
found [here](https://github.com/microsoft/UniSpeech/tree/main/UniSpeech).

## Usage tips

- UniSpeech is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. Please
  use [`Wav2Vec2Processor`] for the feature extraction.
- UniSpeech model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be
  decoded using [`Wav2Vec2CTCTokenizer`].

## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## UniSpeechConfig

[[autodoc]] UniSpeechConfig

## UniSpeech specific outputs

[[autodoc]] models.unispeech.modeling_unispeech.UniSpeechForPreTrainingOutput

## UniSpeechModel

[[autodoc]] UniSpeechModel
    - forward

## UniSpeechForCTC

[[autodoc]] UniSpeechForCTC
    - forward

## UniSpeechForSequenceClassification

[[autodoc]] UniSpeechForSequenceClassification
    - forward

## UniSpeechForPreTraining

[[autodoc]] UniSpeechForPreTraining
    - forward


key: _node_content
string_value:
{"id_": "737a1e69-73ca-400f-8bfc-7523c7f6e321", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cpmant", "title": "CPMAnt", "tokens": 457, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6181e515-85a4-4bf6-9d8c-5cd990d20b71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cpmant", "title": "CPMAnt", "tokens": 457, "retrieve_doc": true, "source": "transformers"}, "hash": "8c816d70be514aa8b620e1fd9bb064a852fd0dfb8b8aeab3320a8d9157c3bdd5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1034, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CPMAnt

## Overview

CPM-Ant is an open-source Chinese pre-trained language model (PLM) with 10B parameters. It is also the first milestone of the live training process of CPM-Live. The training process is cost-effective and environment-friendly. CPM-Ant also achieves promising results with delta tuning on the CUGE benchmark. Besides the full model, we also provide various compressed versions to meet the requirements of different hardware configurations. [See more](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live)

This model was contributed by [OpenBMB](https://huggingface.co/openbmb). The original code can be found [here](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live).

## Resources

- A tutorial on [CPM-Live](https://github.com/OpenBMB/CPM-Live/tree/cpm-ant/cpm-live).

## CpmAntConfig

[[autodoc]] CpmAntConfig
    - all

## CpmAntTokenizer

[[autodoc]] CpmAntTokenizer
    - all

## CpmAntModel

[[autodoc]] CpmAntModel
    - all
    
## CpmAntForCausalLM

[[autodoc]] CpmAntForCausalLM
    - all


key: _node_content
string_value:
{"id_": "3f8eb90a-b638-495f-a0de-4e85bf7424a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd861efa-ed10-4d9b-9a7a-3c4f301de9d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "hash": "9064d64ad12888617726593545d2fda25ddfe35dac01184193068c9d64746c41", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5c63593-0b8a-4916-be5c-a2b7b19144af", "node_type": "1", "metadata": {}, "hash": "806ebcc676b29644c1f0c8f225715545b12e0fa28b80648c10c9618f90dd60ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Speech Encoder Decoder Models

The [`SpeechEncoderDecoderModel`] can be used to initialize a speech-to-text model
with any pretrained speech autoencoding model as the encoder (*e.g.* [Wav2Vec2](wav2vec2), [Hubert](hubert)) and any pretrained autoregressive model as the decoder.

The effectiveness of initializing speech-sequence-to-text-sequence models with pretrained checkpoints for speech
recognition and speech translation has *e.g.* been shown in [Large-Scale Self- and Semi-Supervised Learning for Speech
Translation](https://arxiv.org/abs/2104.06678) by Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli,
Alexis Conneau.

An example of how to use a [`SpeechEncoderDecoderModel`] for inference can be seen in [Speech2Text2](speech_to_text_2).

## Randomly initializing `SpeechEncoderDecoderModel` from model configurations.

[`SpeechEncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`Wav2Vec2Model`] configuration for the encoder
and the default [`BertForCausalLM`] configuration for the decoder.

```python
>>> from transformers import BertConfig, Wav2Vec2Config, SpeechEncoderDecoderConfig, SpeechEncoderDecoderModel

>>> config_encoder = Wav2Vec2Config()
>>> config_decoder = BertConfig()

>>> config = SpeechEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
>>> model = SpeechEncoderDecoderModel(config=config)
```

## Initialising `SpeechEncoderDecoderModel` from a pretrained encoder and a pretrained decoder.

[`SpeechEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based speech model, *e.g.* [Wav2Vec2](wav2vec2), [Hubert](hubert) can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing [`SpeechEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).
To do so, the `SpeechEncoderDecoderModel` class provides a [`SpeechEncoderDecoderModel.from_encoder_decoder_pretrained`] method.

```python
>>> from transformers import SpeechEncoderDecoderModel

>>> model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "facebook/hubert-large-ll60k", "google-bert/bert-base-uncased"
... )
```

## Loading an existing `SpeechEncoderDecoderModel` checkpoint and perform inference.

To load fine-tuned checkpoints of the `SpeechEncoderDecoderModel` class, [`SpeechEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.

To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.


key: _node_content
string_value:
{"id_": "c5c63593-0b8a-4916-be5c-a2b7b19144af", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd861efa-ed10-4d9b-9a7a-3c4f301de9d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "hash": "9064d64ad12888617726593545d2fda25ddfe35dac01184193068c9d64746c41", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f8eb90a-b638-495f-a0de-4e85bf7424a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "hash": "d5cf2faa7114bf450df4ced95be5131b984110f64fd11970365d0570d9084eb2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48808d17-ef6a-4e56-bfea-299d9b7ea253", "node_type": "1", "metadata": {}, "hash": "bb337b06be225d89ce8f742e4eee3192c49066638352bb63d8c29a6cc64d43ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1578, "end_char_idx": 4693, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`SpeechEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based speech model, *e.g.* [Wav2Vec2](wav2vec2), [Hubert](hubert) can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing [`SpeechEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).
To do so, the `SpeechEncoderDecoderModel` class provides a [`SpeechEncoderDecoderModel.from_encoder_decoder_pretrained`] method.

```python
>>> from transformers import SpeechEncoderDecoderModel

>>> model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "facebook/hubert-large-ll60k", "google-bert/bert-base-uncased"
... )
```

## Loading an existing `SpeechEncoderDecoderModel` checkpoint and perform inference.

To load fine-tuned checkpoints of the `SpeechEncoderDecoderModel` class, [`SpeechEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.

To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.

```python
>>> from transformers import Wav2Vec2Processor, SpeechEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> # load a fine-tuned speech translation model and corresponding processor
>>> model = SpeechEncoderDecoderModel.from_pretrained("facebook/wav2vec2-xls-r-300m-en-to-15")
>>> processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-xls-r-300m-en-to-15")

>>> # let's perform inference on a piece of English speech (which we'll translate to German)
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> input_values = processor(ds[0]["audio"]["array"], return_tensors="pt").input_values

>>> # autoregressively generate transcription (uses greedy decoding by default)
>>> generated_ids = model.generate(input_values)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
Mr. Quilter ist der Apostel der Mittelschicht und wir freuen uns, sein Evangelium willkommen heißen zu können.
```

## Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (speech, text) pairs.
As you can see, only 2 inputs are required for the model in order to compute a loss: `input_values` (which are the
speech inputs) and `labels` (which are the `input_ids` of the encoded target sequence).


key: _node_content
string_value:
{"id_": "48808d17-ef6a-4e56-bfea-299d9b7ea253", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd861efa-ed10-4d9b-9a7a-3c4f301de9d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "hash": "9064d64ad12888617726593545d2fda25ddfe35dac01184193068c9d64746c41", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5c63593-0b8a-4916-be5c-a2b7b19144af", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder", "title": "Speech Encoder Decoder Models", "tokens": 1588, "retrieve_doc": true, "source": "transformers"}, "hash": "9e41e421e4d9bfb2f1c78039ae38907364561fb92cd4e8e717f010527c57fe98", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3089, "end_char_idx": 6300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.

```python
>>> from transformers import Wav2Vec2Processor, SpeechEncoderDecoderModel
>>> from datasets import load_dataset
>>> import torch

>>> # load a fine-tuned speech translation model and corresponding processor
>>> model = SpeechEncoderDecoderModel.from_pretrained("facebook/wav2vec2-xls-r-300m-en-to-15")
>>> processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-xls-r-300m-en-to-15")

>>> # let's perform inference on a piece of English speech (which we'll translate to German)
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> input_values = processor(ds[0]["audio"]["array"], return_tensors="pt").input_values

>>> # autoregressively generate transcription (uses greedy decoding by default)
>>> generated_ids = model.generate(input_values)
>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
Mr. Quilter ist der Apostel der Mittelschicht und wir freuen uns, sein Evangelium willkommen heißen zu können.
```

## Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (speech, text) pairs.
As you can see, only 2 inputs are required for the model in order to compute a loss: `input_values` (which are the
speech inputs) and `labels` (which are the `input_ids` of the encoded target sequence).

```python
>>> from transformers import AutoTokenizer, AutoFeatureExtractor, SpeechEncoderDecoderModel
>>> from datasets import load_dataset

>>> encoder_id = "facebook/wav2vec2-base-960h"  # acoustic model encoder
>>> decoder_id = "google-bert/bert-base-uncased"  # text decoder

>>> feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_id)
>>> tokenizer = AutoTokenizer.from_pretrained(decoder_id)
>>> # Combine pre-trained encoder and pre-trained decoder to form a Seq2Seq model
>>> model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_id, decoder_id)

>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
>>> model.config.pad_token_id = tokenizer.pad_token_id

>>> # load an audio input and pre-process (normalise mean/std to 0/1)
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values

>>> # load its corresponding transcription and tokenize to generate labels
>>> labels = tokenizer(ds[0]["text"], return_tensors="pt").input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(input_values=input_values, labels=labels).loss
>>> loss.backward()
```

## SpeechEncoderDecoderConfig

[[autodoc]] SpeechEncoderDecoderConfig

## SpeechEncoderDecoderModel

[[autodoc]] SpeechEncoderDecoderModel
    - forward
    - from_encoder_decoder_pretrained

## FlaxSpeechEncoderDecoderModel

[[autodoc]] FlaxSpeechEncoderDecoderModel
    - __call__
    - from_encoder_decoder_pretrained


key: _node_content
string_value:
{"id_": "0a66c578-9a17-4194-a9ac-f476609e81a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ceb03ce4-9a18-4b8d-a42a-a088a749eafa", "node_type": "1", "metadata": {}, "hash": "e529b13c45eeaa1989b74de85b2102fd684c907ea89523f926d5ad5c752c53ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3006, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MMS

## Overview

The MMS model was proposed in [Scaling Speech Technology to 1,000+ Languages](https://arxiv.org/abs/2305.13516) 
by Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, Michael Auli

The abstract from the paper is the following:

*Expanding the language coverage of speech technology has the potential to improve access to information for many more people. 
However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000
languages spoken around the world. 
The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. 
The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging
self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, 
a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models 
for the same number of languages, as well as a language identification model for 4,017 languages. 
Experiments show that our multilingual speech recognition model more than halves the word error rate of 
Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.*

Here are the different models open sourced in the MMS project. The models and code are originally released [here](https://github.com/facebookresearch/fairseq/tree/main/examples/mms). We have add them to the `transformers` framework, making them easier to use.

### Automatic Speech Recognition (ASR)

The ASR model checkpoints  can be found here : [mms-1b-fl102](https://huggingface.co/facebook/mms-1b-fl102), [mms-1b-l1107](https://huggingface.co/facebook/mms-1b-l1107), [mms-1b-all](https://huggingface.co/facebook/mms-1b-all). For best accuracy, use the `mms-1b-all` model. 

Tips:

- All ASR models accept a float array corresponding to the raw waveform of the speech signal. The raw waveform should be pre-processed with [`Wav2Vec2FeatureExtractor`].
- The models were trained using connectionist temporal classification (CTC) so the model output has to be decoded using
  [`Wav2Vec2CTCTokenizer`].
- You can load different language adapter weights for different languages via [`~Wav2Vec2PreTrainedModel.load_adapter`]. Language adapters only consists of roughly 2 million parameters 
  and can therefore be efficiently loaded on the fly when needed.

#### Loading

By default MMS loads adapter weights for English. If you want to load adapter weights of another language 
make sure to specify `target_lang=<your-chosen-target-lang>` as well as `"ignore_mismatched_sizes=True`.
The `ignore_mismatched_sizes=True` keyword has to be passed to allow the language model head to be resized according
to the vocabulary of the specified language.


key: _node_content
string_value:
{"id_": "ceb03ce4-9a18-4b8d-a42a-a088a749eafa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a66c578-9a17-4194-a9ac-f476609e81a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "0fa998f3692b2c99f101933ef36f7f9c9210f43e16af12526823c56b48469725", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b239e83e-01dd-49f3-825a-93942beac689", "node_type": "1", "metadata": {}, "hash": "9e7cd13ea839e3fdfd8f76b6a01d738cf7d27a7e66fb174d8b6ef0bba58c89ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1459, "end_char_idx": 4514, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

Here are the different models open sourced in the MMS project. The models and code are originally released [here](https://github.com/facebookresearch/fairseq/tree/main/examples/mms). We have add them to the `transformers` framework, making them easier to use.

### Automatic Speech Recognition (ASR)

The ASR model checkpoints  can be found here : [mms-1b-fl102](https://huggingface.co/facebook/mms-1b-fl102), [mms-1b-l1107](https://huggingface.co/facebook/mms-1b-l1107), [mms-1b-all](https://huggingface.co/facebook/mms-1b-all). For best accuracy, use the `mms-1b-all` model. 

Tips:

- All ASR models accept a float array corresponding to the raw waveform of the speech signal. The raw waveform should be pre-processed with [`Wav2Vec2FeatureExtractor`].
- The models were trained using connectionist temporal classification (CTC) so the model output has to be decoded using
  [`Wav2Vec2CTCTokenizer`].
- You can load different language adapter weights for different languages via [`~Wav2Vec2PreTrainedModel.load_adapter`]. Language adapters only consists of roughly 2 million parameters 
  and can therefore be efficiently loaded on the fly when needed.

#### Loading

By default MMS loads adapter weights for English. If you want to load adapter weights of another language 
make sure to specify `target_lang=<your-chosen-target-lang>` as well as `"ignore_mismatched_sizes=True`.
The `ignore_mismatched_sizes=True` keyword has to be passed to allow the language model head to be resized according
to the vocabulary of the specified language.
Similarly, the processor should be loaded with the same target language

```py
from transformers import Wav2Vec2ForCTC, AutoProcessor

model_id = "facebook/mms-1b-all"
target_lang = "fra"

processor = AutoProcessor.from_pretrained(model_id, target_lang=target_lang)
model = Wav2Vec2ForCTC.from_pretrained(model_id, target_lang=target_lang, ignore_mismatched_sizes=True)
```

<Tip>

You can safely ignore a warning such as:

```text
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/mms-1b-all and are newly initialized because the shapes did not match:
- lm_head.bias: found shape torch.Size([154]) in the checkpoint and torch.Size([314]) in the model instantiated
- lm_head.weight: found shape torch.Size([154, 1280]) in the checkpoint and torch.Size([314, 1280]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

</Tip>

If you want to use the ASR pipeline, you can load your chosen target language as such:

```py
from transformers import pipeline

model_id = "facebook/mms-1b-all"
target_lang = "fra"

pipe = pipeline(model=model_id, model_kwargs={"target_lang": "fra", "ignore_mismatched_sizes": True})
```

#### Inference

Next, let's look at how we can run MMS in inference and change adapter layers after having called [`~PretrainedModel.from_pretrained`]
First, we load audio data in different languages using the [Datasets](https://github.com/huggingface/datasets).


key: _node_content
string_value:
{"id_": "b239e83e-01dd-49f3-825a-93942beac689", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ceb03ce4-9a18-4b8d-a42a-a088a749eafa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "45ff2187a6b61160854df9a5ed008684c5a934946c2260785e2ba1e71704f17f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "14e996b3-7e46-40bd-a43e-889c3743247f", "node_type": "1", "metadata": {}, "hash": "da14434dc71b510f4eb6af4105017014096815b855aac5313fd5fda323e50f75", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2845, "end_char_idx": 6000, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `ignore_mismatched_sizes=True` keyword has to be passed to allow the language model head to be resized according
to the vocabulary of the specified language.
Similarly, the processor should be loaded with the same target language

```py
from transformers import Wav2Vec2ForCTC, AutoProcessor

model_id = "facebook/mms-1b-all"
target_lang = "fra"

processor = AutoProcessor.from_pretrained(model_id, target_lang=target_lang)
model = Wav2Vec2ForCTC.from_pretrained(model_id, target_lang=target_lang, ignore_mismatched_sizes=True)
```

<Tip>

You can safely ignore a warning such as:

```text
Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/mms-1b-all and are newly initialized because the shapes did not match:
- lm_head.bias: found shape torch.Size([154]) in the checkpoint and torch.Size([314]) in the model instantiated
- lm_head.weight: found shape torch.Size([154, 1280]) in the checkpoint and torch.Size([314, 1280]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

</Tip>

If you want to use the ASR pipeline, you can load your chosen target language as such:

```py
from transformers import pipeline

model_id = "facebook/mms-1b-all"
target_lang = "fra"

pipe = pipeline(model=model_id, model_kwargs={"target_lang": "fra", "ignore_mismatched_sizes": True})
```

#### Inference

Next, let's look at how we can run MMS in inference and change adapter layers after having called [`~PretrainedModel.from_pretrained`]
First, we load audio data in different languages using the [Datasets](https://github.com/huggingface/datasets).

```py
from datasets import load_dataset, Audio

# English
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]

# French
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "fr", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
fr_sample = next(iter(stream_data))["audio"]["array"]
```

Next, we load the model and processor

```py
from transformers import Wav2Vec2ForCTC, AutoProcessor
import torch

model_id = "facebook/mms-1b-all"

processor = AutoProcessor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id)
```

Now we process the audio data, pass the processed audio data to the model and transcribe the model output,
just like we usually do for [`Wav2Vec2ForCTC`].

```py
inputs = processor(en_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

ids = torch.argmax(outputs, dim=-1)[0]
transcription = processor.decode(ids)
# 'joe keton disapproved of films and buster also had reservations about the media'
```

We can now keep the same model in memory and simply switch out the language adapters by
calling the convenient [`~Wav2Vec2ForCTC.load_adapter`] function for the model and [`~Wav2Vec2CTCTokenizer.set_target_lang`] for the tokenizer.


key: _node_content
string_value:
{"id_": "14e996b3-7e46-40bd-a43e-889c3743247f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b239e83e-01dd-49f3-825a-93942beac689", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "73c99819469de8935973167d8724e8da7484581f95284c7a5a07ba6ffa8f7bc2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4022ee6b-536a-4f88-b1dd-b2ee998d7045", "node_type": "1", "metadata": {}, "hash": "c656f3b8740c8719af7639754c5581ea0baa358736b66cd62e6f39334e7a39a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4516, "end_char_idx": 7556, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from datasets import load_dataset, Audio

# English
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]

# French
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "fr", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
fr_sample = next(iter(stream_data))["audio"]["array"]
```

Next, we load the model and processor

```py
from transformers import Wav2Vec2ForCTC, AutoProcessor
import torch

model_id = "facebook/mms-1b-all"

processor = AutoProcessor.from_pretrained(model_id)
model = Wav2Vec2ForCTC.from_pretrained(model_id)
```

Now we process the audio data, pass the processed audio data to the model and transcribe the model output,
just like we usually do for [`Wav2Vec2ForCTC`].

```py
inputs = processor(en_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

ids = torch.argmax(outputs, dim=-1)[0]
transcription = processor.decode(ids)
# 'joe keton disapproved of films and buster also had reservations about the media'
```

We can now keep the same model in memory and simply switch out the language adapters by
calling the convenient [`~Wav2Vec2ForCTC.load_adapter`] function for the model and [`~Wav2Vec2CTCTokenizer.set_target_lang`] for the tokenizer.
We pass the target language as an input - `"fra"` for French.

```py
processor.tokenizer.set_target_lang("fra")
model.load_adapter("fra")

inputs = processor(fr_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

ids = torch.argmax(outputs, dim=-1)[0]
transcription = processor.decode(ids)
# "ce dernier est volé tout au long de l'histoire romaine"
```

In the same way the language can be switched out for all other supported languages. Please have a look at:

```py
processor.tokenizer.vocab.keys()
```

to see all supported languages.

To further improve performance from ASR models, language model decoding can be used. See the documentation [here](https://huggingface.co/facebook/mms-1b-all) for further details.  

### Speech Synthesis (TTS)

MMS-TTS uses the same model architecture as VITS, which was added to 🤗 Transformers in v4.33. MMS trains a separate 
model checkpoint for each of the 1100+ languages in the project. All available checkpoints can be found on the Hugging 
Face Hub: [facebook/mms-tts](https://huggingface.co/models?sort=trending&search=facebook%2Fmms-tts), and the inference 
documentation under [VITS](https://huggingface.co/docs/transformers/main/en/model_doc/vits).

#### Inference

To use the MMS model, first update to the latest version of the Transformers library:

```bash
pip install --upgrade transformers accelerate
```

Since the flow-based model in VITS is non-deterministic, it is good practice to set a seed to ensure reproducibility of 
the outputs.


key: _node_content
string_value:
{"id_": "4022ee6b-536a-4f88-b1dd-b2ee998d7045", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "14e996b3-7e46-40bd-a43e-889c3743247f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "6e12786d02a195e9fdf561665d46168cb933fe7277aad3581a8dbe8d6cb3e695", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6574dbb3-073c-4f8b-8909-50d4d046e7e7", "node_type": "1", "metadata": {}, "hash": "7063073d6cde19b5b1b1dea12007f459b1c143072d89fe3fb1d62c640cd7e455", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6001, "end_char_idx": 9230, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We pass the target language as an input - `"fra"` for French.

```py
processor.tokenizer.set_target_lang("fra")
model.load_adapter("fra")

inputs = processor(fr_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

ids = torch.argmax(outputs, dim=-1)[0]
transcription = processor.decode(ids)
# "ce dernier est volé tout au long de l'histoire romaine"
```

In the same way the language can be switched out for all other supported languages. Please have a look at:

```py
processor.tokenizer.vocab.keys()
```

to see all supported languages.

To further improve performance from ASR models, language model decoding can be used. See the documentation [here](https://huggingface.co/facebook/mms-1b-all) for further details.  

### Speech Synthesis (TTS)

MMS-TTS uses the same model architecture as VITS, which was added to 🤗 Transformers in v4.33. MMS trains a separate 
model checkpoint for each of the 1100+ languages in the project. All available checkpoints can be found on the Hugging 
Face Hub: [facebook/mms-tts](https://huggingface.co/models?sort=trending&search=facebook%2Fmms-tts), and the inference 
documentation under [VITS](https://huggingface.co/docs/transformers/main/en/model_doc/vits).

#### Inference

To use the MMS model, first update to the latest version of the Transformers library:

```bash
pip install --upgrade transformers accelerate
```

Since the flow-based model in VITS is non-deterministic, it is good practice to set a seed to ensure reproducibility of 
the outputs. 

- For languages with a Roman alphabet, such as English or French, the tokenizer can be used directly to 
pre-process the text inputs. The following code example runs a forward pass using the MMS-TTS English checkpoint:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")

inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")

set_seed(555)  # make deterministic

with torch.no_grad():
   outputs = model(**inputs)

waveform = outputs.waveform[0]
```

The resulting waveform can be saved as a `.wav` file:

```python
import scipy

scipy.io.wavfile.write("synthesized_speech.wav", rate=model.config.sampling_rate, data=waveform)
```

Or displayed in a Jupyter Notebook / Google Colab:

```python
from IPython.display import Audio

Audio(waveform, rate=model.config.sampling_rate)
```

For certain languages with non-Roman alphabets, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) 
perl package is required to pre-process the text inputs to the Roman alphabet.

You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of 
the pre-trained `tokenizer`:

```python
from transformers import VitsTokenizer

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
print(tokenizer.is_uroman)
```

If required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, 
since currently the tokenizer does not support performing the pre-processing itself.


key: _node_content
string_value:
{"id_": "6574dbb3-073c-4f8b-8909-50d4d046e7e7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4022ee6b-536a-4f88-b1dd-b2ee998d7045", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "f53da98448af4f2e0a3f8fd3d9ecb916b5f6210438fd4d41418bc42c07aac0b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1fc3603d-1bb1-4c94-8e4d-3c724b947a31", "node_type": "1", "metadata": {}, "hash": "8036d7e7f1203f73693f30b0bca793c7d885bf407371735cbc736fd969315bc4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7559, "end_char_idx": 10050, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- For languages with a Roman alphabet, such as English or French, the tokenizer can be used directly to 
pre-process the text inputs. The following code example runs a forward pass using the MMS-TTS English checkpoint:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")

inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")

set_seed(555)  # make deterministic

with torch.no_grad():
   outputs = model(**inputs)

waveform = outputs.waveform[0]
```

The resulting waveform can be saved as a `.wav` file:

```python
import scipy

scipy.io.wavfile.write("synthesized_speech.wav", rate=model.config.sampling_rate, data=waveform)
```

Or displayed in a Jupyter Notebook / Google Colab:

```python
from IPython.display import Audio

Audio(waveform, rate=model.config.sampling_rate)
```

For certain languages with non-Roman alphabets, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) 
perl package is required to pre-process the text inputs to the Roman alphabet.

You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of 
the pre-trained `tokenizer`:

```python
from transformers import VitsTokenizer

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
print(tokenizer.is_uroman)
```

If required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, 
since currently the tokenizer does not support performing the pre-processing itself.

To do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:

```bash
git clone https://github.com/isi-nlp/uroman.git
cd uroman
export UROMAN=$(pwd)
```

You can then pre-process the text input using the following code snippet. You can either rely on using the bash variable 
`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed
import os
import subprocess

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-kor")
model = VitsModel.from_pretrained("facebook/mms-tts-kor")

def uromanize(input_string, uroman_path):
    """Convert non-Roman strings to Roman using the `uroman` perl package."""


key: _node_content
string_value:
{"id_": "1fc3603d-1bb1-4c94-8e4d-3c724b947a31", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6574dbb3-073c-4f8b-8909-50d4d046e7e7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "29c5dd8807c63ad99692b3592d319c20265dd7f1492339f7e57f3e6330000283", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d66d2b52-886a-4f61-a4c8-de2ffbb25327", "node_type": "1", "metadata": {}, "hash": "9b1ca43ee45fbe8d3a455ed8092172510bd5ef0a143cc283f8ab41d73eedb337", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8731, "end_char_idx": 11466, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of 
the pre-trained `tokenizer`:

```python
from transformers import VitsTokenizer

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
print(tokenizer.is_uroman)
```

If required, you should apply the uroman package to your text inputs **prior** to passing them to the `VitsTokenizer`, 
since currently the tokenizer does not support performing the pre-processing itself.

To do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:

```bash
git clone https://github.com/isi-nlp/uroman.git
cd uroman
export UROMAN=$(pwd)
```

You can then pre-process the text input using the following code snippet. You can either rely on using the bash variable 
`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed
import os
import subprocess

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-kor")
model = VitsModel.from_pretrained("facebook/mms-tts-kor")

def uromanize(input_string, uroman_path):
    """Convert non-Roman strings to Roman using the `uroman` perl package."""
    script_path = os.path.join(uroman_path, "bin", "uroman.pl")

    command = ["perl", script_path]

    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # Execute the perl command
    stdout, stderr = process.communicate(input=input_string.encode())

    if process.returncode != 0:
        raise ValueError(f"Error {process.returncode}: {stderr.decode()}")

    # Return the output as a string and skip the new-line character at the end
    return stdout.decode()[:-1]

text = "이봐 무슨 일이야"
uromaized_text = uromanize(text, uroman_path=os.environ["UROMAN"])

inputs = tokenizer(text=uromaized_text, return_tensors="pt")

set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(inputs["input_ids"])

waveform = outputs.waveform[0]
```

**Tips:**

* The MMS-TTS checkpoints are trained on lower-cased, un-punctuated text. By default, the `VitsTokenizer` *normalizes* the inputs by removing any casing and punctuation, to avoid passing out-of-vocabulary characters to the model. Hence, the model is agnostic to casing and punctuation, so these should be avoided in the text prompt. You can disable normalisation by setting `normalize=False` in the call to the tokenizer, but this will lead to un-expected behaviour and is discouraged.
* The speaking rate can be varied by setting the attribute `model.speaking_rate` to a chosen value.


key: _node_content
string_value:
{"id_": "d66d2b52-886a-4f61-a4c8-de2ffbb25327", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1fc3603d-1bb1-4c94-8e4d-3c724b947a31", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "0c22cce6efc0cf4c1391cb2f9555a2163ffd59bcafd50b62521aaa3b7194409e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00e863f0-d153-4d6b-8aeb-766ee41f8afe", "node_type": "1", "metadata": {}, "hash": "b9a6cbc717434a4c650dcbda78e7ba5dc949295bd46e1dec5c919e0d4d879015", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10055, "end_char_idx": 12726, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
script_path = os.path.join(uroman_path, "bin", "uroman.pl")

    command = ["perl", script_path]

    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # Execute the perl command
    stdout, stderr = process.communicate(input=input_string.encode())

    if process.returncode != 0:
        raise ValueError(f"Error {process.returncode}: {stderr.decode()}")

    # Return the output as a string and skip the new-line character at the end
    return stdout.decode()[:-1]

text = "이봐 무슨 일이야"
uromaized_text = uromanize(text, uroman_path=os.environ["UROMAN"])

inputs = tokenizer(text=uromaized_text, return_tensors="pt")

set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(inputs["input_ids"])

waveform = outputs.waveform[0]
```

**Tips:**

* The MMS-TTS checkpoints are trained on lower-cased, un-punctuated text. By default, the `VitsTokenizer` *normalizes* the inputs by removing any casing and punctuation, to avoid passing out-of-vocabulary characters to the model. Hence, the model is agnostic to casing and punctuation, so these should be avoided in the text prompt. You can disable normalisation by setting `normalize=False` in the call to the tokenizer, but this will lead to un-expected behaviour and is discouraged.
* The speaking rate can be varied by setting the attribute `model.speaking_rate` to a chosen value. Likewise, the randomness of the noise is controlled by `model.noise_scale`:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")

inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")

# make deterministic
set_seed(555)  

# make speech faster and more noisy
model.speaking_rate = 1.5
model.noise_scale = 0.8

with torch.no_grad():
   outputs = model(**inputs)
```

### Language Identification (LID)

Different LID models are available based on the number of languages they can recognize - [126](https://huggingface.co/facebook/mms-lid-126), [256](https://huggingface.co/facebook/mms-lid-256), [512](https://huggingface.co/facebook/mms-lid-512), [1024](https://huggingface.co/facebook/mms-lid-1024), [2048](https://huggingface.co/facebook/mms-lid-2048), [4017](https://huggingface.co/facebook/mms-lid-4017). 

#### Inference
First, we install transformers and some other libraries

```bash
pip install torch accelerate datasets[audio]
pip install --upgrade transformers
````

Next, we load a couple of audio samples via `datasets`. Make sure that the audio data is sampled to 16000 kHz.


key: _node_content
string_value:
{"id_": "00e863f0-d153-4d6b-8aeb-766ee41f8afe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d66d2b52-886a-4f61-a4c8-de2ffbb25327", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "5c2976fbb2ba91f62af3ae00dd3c04860f7be27915125537d022956cbdae2108", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a45a6e02-1257-4294-89df-85df51ffccba", "node_type": "1", "metadata": {}, "hash": "9ac2dad7ed6dc6356ab78625514b54c459e219990aa148fcd2d0dfd0e0352dd4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11214, "end_char_idx": 14720, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can disable normalisation by setting `normalize=False` in the call to the tokenizer, but this will lead to un-expected behaviour and is discouraged.
* The speaking rate can be varied by setting the attribute `model.speaking_rate` to a chosen value. Likewise, the randomness of the noise is controlled by `model.noise_scale`:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")

inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")

# make deterministic
set_seed(555)  

# make speech faster and more noisy
model.speaking_rate = 1.5
model.noise_scale = 0.8

with torch.no_grad():
   outputs = model(**inputs)
```

### Language Identification (LID)

Different LID models are available based on the number of languages they can recognize - [126](https://huggingface.co/facebook/mms-lid-126), [256](https://huggingface.co/facebook/mms-lid-256), [512](https://huggingface.co/facebook/mms-lid-512), [1024](https://huggingface.co/facebook/mms-lid-1024), [2048](https://huggingface.co/facebook/mms-lid-2048), [4017](https://huggingface.co/facebook/mms-lid-4017). 

#### Inference
First, we install transformers and some other libraries

```bash
pip install torch accelerate datasets[audio]
pip install --upgrade transformers
````

Next, we load a couple of audio samples via `datasets`. Make sure that the audio data is sampled to 16000 kHz.

```py
from datasets import load_dataset, Audio

# English
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]

# Arabic
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "ar", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
ar_sample = next(iter(stream_data))["audio"]["array"]
```

Next, we load the model and processor

```py
from transformers import Wav2Vec2ForSequenceClassification, AutoFeatureExtractor
import torch

model_id = "facebook/mms-lid-126"

processor = AutoFeatureExtractor.from_pretrained(model_id)
model = Wav2Vec2ForSequenceClassification.from_pretrained(model_id)
```

Now we process the audio data, pass the processed audio data to the model to classify it into a language, just like we usually do for Wav2Vec2 audio classification models such as [ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition](https://huggingface.co/harshit345/xlsr-wav2vec-speech-emotion-recognition)

```py
# English
inputs = processor(en_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

lang_id = torch.argmax(outputs, dim=-1)[0].item()
detected_lang = model.config.id2label[lang_id]
# 'eng'

# Arabic
inputs = processor(ar_sample, sampling_rate=16_000, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs).logits

lang_id = torch.argmax(outputs, dim=-1)[0].item()
detected_lang = model.config.id2label[lang_id]
# 'ara'
```

To see all the supported languages of a checkpoint, you can print out the language ids as follows:
```py
processor.id2label.values()
```

### Audio Pretrained Models

Pretrained models are available for two different sizes - [300M](https://huggingface.co/facebook/mms-300m) , 
[1Bil](https://huggingface.co/facebook/mms-1b).


key: _node_content
string_value:
{"id_": "a45a6e02-1257-4294-89df-85df51ffccba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb6d18c1-4c4f-4154-bbbf-320d598899cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "cd807bd8feda6576508976b2a8c8027d1bcbd01e4f28157cb9be5ccc8324b07d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00e863f0-d153-4d6b-8aeb-766ee41f8afe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mms", "title": "MMS", "tokens": 3875, "retrieve_doc": true, "source": "transformers"}, "hash": "0035294a3a8b4a6bf18d0a0b365fa01060fe5daf7e36837404105c1a985c96d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14723, "end_char_idx": 15043, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

The MMS for ASR architecture is based on the Wav2Vec2 model, refer to [Wav2Vec2's documentation page](wav2vec2) for further 
details on how to finetune with models for various downstream tasks.

MMS-TTS uses the same model architecture as VITS, refer to [VITS's documentation page](vits) for API reference.
</Tip>


key: _node_content
string_value:
{"id_": "739f9657-5094-420e-aeeb-c7599d35d02e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bort", "title": "BORT", "tokens": 785, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7fe28a6c-0310-4f0a-bf7a-c945117c968b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bort", "title": "BORT", "tokens": 785, "retrieve_doc": true, "source": "transformers"}, "hash": "4eee076ae2b828a229d342a7a737ab2bfefc4767b9c6e3bdbba97e1943f10975", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BORT

<Tip warning={true}>

This model is in maintenance mode only, we do not accept any new PRs changing its code.

If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.
You can do so by running the following command: `pip install -U transformers==4.30.0`.

</Tip>

## Overview

The BORT model was proposed in [Optimal Subarchitecture Extraction for BERT](https://arxiv.org/abs/2010.10499) by
Adrian de Wynter and Daniel J. Perry. It is an optimal subset of architectural parameters for the BERT, which the
authors refer to as "Bort".

The abstract from the paper is the following:

*We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by
applying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as
"Bort", is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of 5.5% the
original BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in 288 GPU hours, which
is 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large
(Liu et al., 2019), and about 33% of that of the world-record, in GPU hours, required to train BERT-large on the same
hardware. It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the
architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%,
absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.*

This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/alexa/bort/).

## Usage tips

- BORT's model architecture is based on BERT, refer to [BERT's documentation page](bert) for the
  model's API reference as well as usage examples.
- BORT uses the RoBERTa tokenizer instead of the BERT tokenizer, refer to [RoBERTa's documentation page](roberta) for the tokenizer's API reference as well as usage examples.
- BORT requires a specific fine-tuning algorithm, called [Agora](https://adewynter.github.io/notes/bort_algorithms_and_applications.html#fine-tuning-with-algebraic-topology) ,
  that is sadly not open-sourced yet. It would be very useful for the community, if someone tries to implement the
  algorithm to make BORT fine-tuning work.


key: _node_content
string_value:
{"id_": "0f40f0cc-59e5-4959-81bc-92d314e8ea59", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31739a47-60c9-4d63-8849-462b8bc44893", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "transformers"}, "hash": "ed1501d4df450a2a72dfb3140d7c561a37c594c8a324852dd7030619db224f1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f5d989c-1666-42d5-bbc1-98d4c3398b98", "node_type": "1", "metadata": {}, "hash": "d4f18e94a5fd48abc78fa694c50cb084e2c9d6758db135cb3c69da09634a1eb2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2625, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SqueezeBERT

## Overview

The SqueezeBERT model was proposed in [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://arxiv.org/abs/2006.11316) by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer. It's a
bidirectional transformer similar to the BERT model. The key difference between the BERT architecture and the
SqueezeBERT architecture is that SqueezeBERT uses [grouped convolutions](https://blog.yani.io/filter-group-tutorial)
instead of fully-connected layers for the Q, K, V and FFN layers.

The abstract from the paper is the following:

*Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets,
large computing systems, and better neural network models, natural language processing (NLP) technology has made
significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant
opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we
consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's
highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with
BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods
such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these
techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in
self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called
SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test
set. The SqueezeBERT code will be released.*

This model was contributed by [forresti](https://huggingface.co/forresti).

## Usage tips

- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right
  rather than the left.
- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
  with a causal language modeling (CLM) objective are better in that regard.
- For best results when finetuning on sequence classification tasks, it is recommended to start with the
  *squeezebert/squeezebert-mnli-headless* checkpoint.


key: _node_content
string_value:
{"id_": "6f5d989c-1666-42d5-bbc1-98d4c3398b98", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31739a47-60c9-4d63-8849-462b8bc44893", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "transformers"}, "hash": "ed1501d4df450a2a72dfb3140d7c561a37c594c8a324852dd7030619db224f1a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f40f0cc-59e5-4959-81bc-92d314e8ea59", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/squeezebert", "title": "SqueezeBERT", "tokens": 1032, "retrieve_doc": true, "source": "transformers"}, "hash": "a6ef6080f53f9378322499d646137851ea9fecdb642cb0305c7c7365cf4f5bc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 916, "end_char_idx": 3730, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Thus, there is a significant
opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we
consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's
highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with
BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods
such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these
techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in
self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called
SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test
set. The SqueezeBERT code will be released.*

This model was contributed by [forresti](https://huggingface.co/forresti).

## Usage tips

- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right
  rather than the left.
- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
  with a causal language modeling (CLM) objective are better in that regard.
- For best results when finetuning on sequence classification tasks, it is recommended to start with the
  *squeezebert/squeezebert-mnli-headless* checkpoint.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## SqueezeBertConfig

[[autodoc]] SqueezeBertConfig

## SqueezeBertTokenizer

[[autodoc]] SqueezeBertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## SqueezeBertTokenizerFast

[[autodoc]] SqueezeBertTokenizerFast

## SqueezeBertModel

[[autodoc]] SqueezeBertModel

## SqueezeBertForMaskedLM

[[autodoc]] SqueezeBertForMaskedLM

## SqueezeBertForSequenceClassification

[[autodoc]] SqueezeBertForSequenceClassification

## SqueezeBertForMultipleChoice

[[autodoc]] SqueezeBertForMultipleChoice

## SqueezeBertForTokenClassification

[[autodoc]] SqueezeBertForTokenClassification

## SqueezeBertForQuestionAnswering

[[autodoc]] SqueezeBertForQuestionAnswering


key: _node_content
string_value:
{"id_": "718dbe15-74a6-4f99-a8ab-fc39d30491c4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2_phoneme", "title": "Wav2Vec2Phoneme", "tokens": 748, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "03a02368-55ce-4041-a1d8-4974a2491f43", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2_phoneme", "title": "Wav2Vec2Phoneme", "tokens": 748, "retrieve_doc": true, "source": "transformers"}, "hash": "0ed977ba2d71e44f1157cb6a400536840b5cafa3ee3092b33de51e1011119800", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2315, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Wav2Vec2Phoneme

## Overview

The Wav2Vec2Phoneme model was proposed in [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition (Xu et al.,
2021](https://arxiv.org/abs/2109.11680) by Qiantong Xu, Alexei Baevski, Michael Auli.

The abstract from the paper is the following:

*Recent progress in self-training, self-supervised pretraining and unsupervised learning enabled well performing speech
recognition systems without any labeled data. However, in many cases there is labeled data available for related
languages which is not utilized by these methods. This paper extends previous work on zero-shot cross-lingual transfer
learning by fine-tuning a multilingually pretrained wav2vec 2.0 model to transcribe unseen languages. This is done by
mapping phonemes of the training languages to the target language using articulatory features. Experiments show that
this simple method significantly outperforms prior work which introduced task-specific architectures and used only part
of a monolingually pretrained model.*

Relevant checkpoints can be found under https://huggingface.co/models?other=phoneme-recognition.

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten)

The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).

## Usage tips

- Wav2Vec2Phoneme uses the exact same architecture as Wav2Vec2
- Wav2Vec2Phoneme is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- Wav2Vec2Phoneme model was trained using connectionist temporal classification (CTC) so the model output has to be
  decoded using [`Wav2Vec2PhonemeCTCTokenizer`].
- Wav2Vec2Phoneme can be fine-tuned on multiple language at once and decode unseen languages in a single forward pass
  to a sequence of phonemes
- By default, the model outputs a sequence of phonemes. In order to transform the phonemes to a sequence of words one
  should make use of a dictionary and language model.


<Tip>

Wav2Vec2Phoneme's architecture is based on the Wav2Vec2 model, for API reference, check out [`Wav2Vec2`](wav2vec2)'s documentation page 
except for the tokenizer.

</Tip>

## Wav2Vec2PhonemeCTCTokenizer

[[autodoc]] Wav2Vec2PhonemeCTCTokenizer
	- __call__
	- batch_decode
	- decode
	- phonemize


key: _node_content
string_value:
{"id_": "0168c1cf-d2c1-451d-a5f9-33c5306bcf8e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/big_bird", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1e76bfec-b808-4896-b86e-1c45a0eb74d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/big_bird", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "transformers"}, "hash": "320b69cc5256101324a378d3f9852df447edbc4ff51bf610e516d673ca325629", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "36aeb195-1789-4d04-b60f-d9f15711c53f", "node_type": "1", "metadata": {}, "hash": "b616d1f7fa0cf21b7fbac73ec7fa57d7537032c3d7d6c460588e2fe5fee80246", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3017, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BigBird

## Overview

The BigBird model was proposed in [Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062) by
Zaheer, Manzil and Guruganesh, Guru and Dubey, Kumar Avinava and Ainslie, Joshua and Alberti, Chris and Ontanon,
Santiago and Pham, Philip and Ravula, Anirudh and Wang, Qifan and Yang, Li and others. BigBird, is a sparse-attention
based transformer which extends Transformer based models, such as BERT to much longer sequences. In addition to sparse
attention, BigBird also applies global attention as well as random attention to the input sequence. Theoretically, it
has been shown that applying sparse, global, and random attention approximates full attention, while being
computationally much more efficient for longer sequences. As a consequence of the capability to handle longer context,
BigBird has shown improved performance on various long document NLP tasks, such as question answering and
summarization, compared to BERT or RoBERTa.

The abstract from the paper is the following:

*Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP.
Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence
length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that
reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and
is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our
theoretical analysis reveals some of the benefits of having O(1) global tokens (such as CLS), that attend to the entire
sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to
8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context,
BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also
propose novel applications to genomics data.*

This model was contributed by [vasudevgupta](https://huggingface.co/vasudevgupta). The original code can be found
[here](https://github.com/google-research/bigbird).

## Usage tips

- For an in-detail explanation on how BigBird's attention works, see [this blog post](https://huggingface.co/blog/big-bird).
- BigBird comes with 2 implementations: **original_full** & **block_sparse**. For the sequence length < 1024, using
  **original_full** is advised as there is no benefit in using **block_sparse** attention.
- The code currently uses window size of 3 blocks and 2 global blocks.
- Sequence length must be divisible by block size.
- Current implementation supports only **ITC**.
- Current implementation doesn't support **num_random_blocks = 0**
- BigBird is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.


key: _node_content
string_value:
{"id_": "36aeb195-1789-4d04-b60f-d9f15711c53f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/big_bird", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1e76bfec-b808-4896-b86e-1c45a0eb74d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/big_bird", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "transformers"}, "hash": "320b69cc5256101324a378d3f9852df447edbc4ff51bf610e516d673ca325629", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0168c1cf-d2c1-451d-a5f9-33c5306bcf8e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/big_bird", "title": "BigBird", "tokens": 1440, "retrieve_doc": true, "source": "transformers"}, "hash": "fb87f2e82df3baf13efebacbb726986efe30f993510d74adfc19650a1c980008", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3020, "end_char_idx": 5215, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## BigBirdConfig

[[autodoc]] BigBirdConfig

## BigBirdTokenizer

[[autodoc]] BigBirdTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## BigBirdTokenizerFast

[[autodoc]] BigBirdTokenizerFast

## BigBird specific outputs

[[autodoc]] models.big_bird.modeling_big_bird.BigBirdForPreTrainingOutput

<frameworkcontent>
<pt>

## BigBirdModel

[[autodoc]] BigBirdModel
    - forward

## BigBirdForPreTraining

[[autodoc]] BigBirdForPreTraining
    - forward

## BigBirdForCausalLM

[[autodoc]] BigBirdForCausalLM
    - forward

## BigBirdForMaskedLM

[[autodoc]] BigBirdForMaskedLM
    - forward

## BigBirdForSequenceClassification

[[autodoc]] BigBirdForSequenceClassification
    - forward

## BigBirdForMultipleChoice

[[autodoc]] BigBirdForMultipleChoice
    - forward

## BigBirdForTokenClassification

[[autodoc]] BigBirdForTokenClassification
    - forward

## BigBirdForQuestionAnswering

[[autodoc]] BigBirdForQuestionAnswering
    - forward

</pt>
<jax>

## FlaxBigBirdModel

[[autodoc]] FlaxBigBirdModel
    - __call__

## FlaxBigBirdForPreTraining

[[autodoc]] FlaxBigBirdForPreTraining
    - __call__

## FlaxBigBirdForCausalLM

[[autodoc]] FlaxBigBirdForCausalLM
    - __call__

## FlaxBigBirdForMaskedLM

[[autodoc]] FlaxBigBirdForMaskedLM
    - __call__

## FlaxBigBirdForSequenceClassification

[[autodoc]] FlaxBigBirdForSequenceClassification
    - __call__

## FlaxBigBirdForMultipleChoice

[[autodoc]] FlaxBigBirdForMultipleChoice
    - __call__

## FlaxBigBirdForTokenClassification

[[autodoc]] FlaxBigBirdForTokenClassification
    - __call__

## FlaxBigBirdForQuestionAnswering

[[autodoc]] FlaxBigBirdForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "927e9dbd-6afd-4f6f-8b5e-a23dc9958556", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5057eef9-5e66-44c7-9608-2a75e460a42d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0c37a9e5eadad6fb0406ca40f3e579b282df4165dcf81ccd59c7b13a46522021", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ad54645-fcdc-440d-a66f-a2be2ea62fc5", "node_type": "1", "metadata": {}, "hash": "dcea0642d00f5f4cf469a1f4c002db384b3fa7033bde79248802b1856a7ca7fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2872, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Mixtral

## Overview

Mixtral-8x7B was introduced in the [Mixtral of Experts blogpost](https://mistral.ai/news/mixtral-of-experts/) by Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed.

The introduction of the blog post says:

*Today, the team is proud to release Mixtral 8x7B, a high-quality sparse mixture of experts models (SMoE) with open weights. Licensed under Apache 2.0. Mixtral outperforms Llama 2 70B on most benchmarks with 6x faster inference. It is the strongest open-weight model with a permissive license and the best model overall regarding cost/performance trade-offs. In particular, it matches or outperforms GPT3.5 on most standard benchmarks.*

Mixtral-8x7B is the second large language model (LLM) released by [mistral.ai](https://mistral.ai/), after [Mistral-7B](mistral).

### Architectural details

Mixtral-8x7B is a decoder-only Transformer with the following architectural choices:

- Mixtral is a Mixture of Experts (MoE) model with 8 experts per MLP, with a total of 45 billion parameters. To learn more about mixture-of-experts, refer to the [blog post](https://huggingface.co/blog/moe).
- Despite the model having 45 billion parameters,, the compute required for a single forward pass is the same as that of a 14 billion parameter model. This is because even though each of the experts have to be loaded in RAM (70B like ram requirement) each token from the hidden states are dispatched twice (top 2 routing) and thus the compute (the operation required at each forward computation) is just 2 X sequence_length. 

The following implementation details are shared with Mistral AI's first model [Mistral-7B](mistral):
- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens
- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.
- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.

For more details refer to the [release blog post](https://mistral.ai/news/mixtral-of-experts/).

### License

`Mixtral-8x7B` is released under the Apache 2.0 license.

## Usage tips

The Mistral team has released 2 checkpoints:
- a base model, [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.
- an instruction tuned model, [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).


key: _node_content
string_value:
{"id_": "4ad54645-fcdc-440d-a66f-a2be2ea62fc5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5057eef9-5e66-44c7-9608-2a75e460a42d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0c37a9e5eadad6fb0406ca40f3e579b282df4165dcf81ccd59c7b13a46522021", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "927e9dbd-6afd-4f6f-8b5e-a23dc9958556", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "c7063e7f96ac5d774a037f290208a25b940e1db6382669114cc02ad4722deac4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b4ade85-9f1f-46d2-8deb-c20275089e76", "node_type": "1", "metadata": {}, "hash": "48a11e7c30c0c76b6e660ba8dbbb71198448ada8df9f4158ed1bd0b81f483762", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1385, "end_char_idx": 4158, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Despite the model having 45 billion parameters,, the compute required for a single forward pass is the same as that of a 14 billion parameter model. This is because even though each of the experts have to be loaded in RAM (70B like ram requirement) each token from the hidden states are dispatched twice (top 2 routing) and thus the compute (the operation required at each forward computation) is just 2 X sequence_length. 

The following implementation details are shared with Mistral AI's first model [Mistral-7B](mistral):
- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens
- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.
- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.

For more details refer to the [release blog post](https://mistral.ai/news/mixtral-of-experts/).

### License

`Mixtral-8x7B` is released under the Apache 2.0 license.

## Usage tips

The Mistral team has released 2 checkpoints:
- a base model, [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1), which has been pre-trained to predict the next token on internet-scale data.
- an instruction tuned model, [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1), which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).

The base model can be used as follows:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1")

>>> prompt = "My favourite condiment is"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"My favourite condiment is to ..."
```

The instruction tuned model can be used as follows:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")

>>> messages = [
...     {"role": "user", "content": "What is your favourite condiment?"},
...     {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
...     {"role": "user", "content": "Do you have mayonnaise recipes?"}


key: _node_content
string_value:
{"id_": "9b4ade85-9f1f-46d2-8deb-c20275089e76", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5057eef9-5e66-44c7-9608-2a75e460a42d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0c37a9e5eadad6fb0406ca40f3e579b282df4165dcf81ccd59c7b13a46522021", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ad54645-fcdc-440d-a66f-a2be2ea62fc5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0525f7194c4d9f376068267be3290e4ba8a7e0dbba37344c69f344d309bac7ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a7aa77c4-02db-40fb-909f-47c9b0e1ff67", "node_type": "1", "metadata": {}, "hash": "b571ec0696e6d767c15269ed0be8d0fe530c73eef9db80d77e0745d5fa70c7bf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2874, "end_char_idx": 5384, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The base model can be used as follows:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1")

>>> prompt = "My favourite condiment is"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"My favourite condiment is to ..."
```

The instruction tuned model can be used as follows:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")

>>> messages = [
...     {"role": "user", "content": "What is your favourite condiment?"},
...     {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
...     {"role": "user", "content": "Do you have mayonnaise recipes?"}
... ]

>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"Mayonnaise can be made as follows: (...)"
```

As can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.

## Speeding up Mixtral by using Flash Attention

The code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g.


key: _node_content
string_value:
{"id_": "a7aa77c4-02db-40fb-909f-47c9b0e1ff67", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5057eef9-5e66-44c7-9608-2a75e460a42d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0c37a9e5eadad6fb0406ca40f3e579b282df4165dcf81ccd59c7b13a46522021", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b4ade85-9f1f-46d2-8deb-c20275089e76", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "59870213f60536ff5162fccea515bfa11b357886ed57b9b2b88885aac64f9cd8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2b88cbc-998d-43e5-88c0-64b24f54183c", "node_type": "1", "metadata": {}, "hash": "0a88a5c1f38288a1bc35d318b2b7f978f6ab611e6623e8a5008504232ae6ce9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3885, "end_char_idx": 7200, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
...     {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
...     {"role": "user", "content": "Do you have mayonnaise recipes?"}
... ]

>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"Mayonnaise can be made as follows: (...)"
```

As can be seen, the instruction-tuned model requires a [chat template](../chat_templating) to be applied to make sure the inputs are prepared in the right format.

## Speeding up Mixtral by using Flash Attention

The code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)

To load and run a model using Flash Attention-2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1", torch_dtype=torch.float16, attn_implementation="flash_attention_2", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1")

>>> prompt = "My favourite condiment is"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"The expected output"
```

### Expected speedups

Below is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `mistralai/Mixtral-8x7B-v0.1` checkpoint and the Flash Attention 2 version of the model.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mixtral-7b-inference-large-seqlen.png">
</div>

### Sliding window Attention

The current implementation supports the sliding window attention mechanism and memory efficient cache management. 
To enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). 

The Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side="left"` and use the absolute position of the current token to compute the positional embedding.


key: _node_content
string_value:
{"id_": "d2b88cbc-998d-43e5-88c0-64b24f54183c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5057eef9-5e66-44c7-9608-2a75e460a42d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0c37a9e5eadad6fb0406ca40f3e579b282df4165dcf81ccd59c7b13a46522021", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a7aa77c4-02db-40fb-909f-47c9b0e1ff67", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "9421fc1234d1d1d77bb62627ff5c740cc6fc42d24b45114f7d2e60c0c5feff65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bfb7dcb3-93fe-446a-bb21-3df81e27cec4", "node_type": "1", "metadata": {}, "hash": "957be057f0f06ad3fa9d2efe0777af5b226d5de8bf362b27111e1582957176d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6352, "end_char_idx": 9452, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/mixtral-7b-inference-large-seqlen.png">
</div>

### Sliding window Attention

The current implementation supports the sliding window attention mechanism and memory efficient cache management. 
To enable sliding window attention, just make sure to have a `flash-attn` version that is compatible with sliding window attention (`>=2.3.0`). 

The Flash Attention-2 model uses also a more memory efficient cache slicing mechanism - as recommended per the official implementation of Mistral model that use rolling cache mechanism we keep the cache size fixed (`self.config.sliding_window`), support batched generation only for `padding_side="left"` and use the absolute position of the current token to compute the positional embedding.

## Shrinking down Mixtral using quantization

As the Mixtral model has 45 billion parameters, that would require about 90GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), a single A100 with 40GB of RAM is enough to fit the entire model, as in that case only about 27 GB of RAM is required.

Quantizing a model is as simple as passing a `quantization_config` to the model. Below, we'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

>>> # specify how to quantize the model
>>> quantization_config = BitsAndBytesConfig(
...         load_in_4bit=True,
...         bnb_4bit_quant_type="nf4",
...         bnb_4bit_compute_dtype="torch.float16",
... )

>>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1", quantization_config=True, device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")

>>> prompt = "My favourite condiment is"

>>> messages = [
...     {"role": "user", "content": "What is your favourite condiment?"},
...     {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
...     {"role": "user", "content": "Do you have mayonnaise recipes?"}
... ]

>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"The expected output"
```

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .
The original code can be found [here](https://github.com/mistralai/mistral-src).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Mixtral.


key: _node_content
string_value:
{"id_": "bfb7dcb3-93fe-446a-bb21-3df81e27cec4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5057eef9-5e66-44c7-9608-2a75e460a42d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0c37a9e5eadad6fb0406ca40f3e579b282df4165dcf81ccd59c7b13a46522021", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2b88cbc-998d-43e5-88c0-64b24f54183c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mixtral", "title": "Mixtral", "tokens": 2917, "retrieve_doc": true, "source": "transformers"}, "hash": "0695fd61d24a70bc75514af71beaa3e2792c4799cbb2dbbef4a7af971b9b1029", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8582, "end_char_idx": 10843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
...     {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
...     {"role": "user", "content": "Do you have mayonnaise recipes?"}
... ]

>>> model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

>>> generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"The expected output"
```

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .
The original code can be found [here](https://github.com/mistralai/mistral-src).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Mixtral. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-generation"/>

- A demo notebook to perform supervised fine-tuning (SFT) of Mixtral-8x7B can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb). 🌎
- A [blog post](https://medium.com/@prakharsaxena11111/finetuning-mixtral-7bx8-6071b0ebf114) on fine-tuning Mixtral-8x7B using PEFT. 🌎
- The [Alignment Handbook](https://github.com/huggingface/alignment-handbook) by Hugging Face includes scripts and recipes to perform supervised fine-tuning (SFT) and direct preference optimization with Mistral-7B. This includes scripts for full fine-tuning, QLoRa on a single GPU as well as multi-GPU fine-tuning.
- [Causal language modeling task guide](../tasks/language_modeling)

## MixtralConfig

[[autodoc]] MixtralConfig

## MixtralModel

[[autodoc]] MixtralModel
    - forward

## MixtralForCausalLM

[[autodoc]] MixtralForCausalLM
    - forward

## MixtralForSequenceClassification

[[autodoc]] MixtralForSequenceClassification
    - forward

## MixtralForTokenClassification

[[autodoc]] MixtralForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "4d20af1d-52ce-49dc-8b98-352a7592861c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/graphormer", "title": "Graphormer", "tokens": 737, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb73c656-1fc0-4dd0-a31b-470357e96bd8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/graphormer", "title": "Graphormer", "tokens": 737, "retrieve_doc": true, "source": "transformers"}, "hash": "52687ebe2185163437af1864a86da63f86bfd0c6630a514d6a3f53bfca343e2e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2780, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Graphormer

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The Graphormer model was proposed in [Do Transformers Really Perform Bad for Graph Representation?](https://arxiv.org/abs/2106.05234)  by
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen and Tie-Yan Liu. It is a Graph Transformer model, modified to allow computations on graphs instead of text sequences by generating embeddings and features of interest during preprocessing and collation, then using a modified attention.

The abstract from the paper is the following:

*The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level prediction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured data. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer.*

This model was contributed by [clefourrier](https://huggingface.co/clefourrier). The original code can be found [here](https://github.com/microsoft/Graphormer).

## Usage tips

This model will not work well on large graphs (more than 100 nodes/edges), as it will make the memory explode.
You can reduce the batch size, increase your RAM, or decrease the `UNREACHABLE_NODE_DISTANCE` parameter in algos_graphormer.pyx, but it will be hard to go above 700 nodes/edges.

This model does not use a tokenizer, but instead a special collator during training.

## GraphormerConfig

[[autodoc]] GraphormerConfig

## GraphormerModel

[[autodoc]] GraphormerModel
    - forward

## GraphormerForGraphClassification

[[autodoc]] GraphormerForGraphClassification
    - forward


key: _node_content
string_value:
{"id_": "88fb8331-2e21-4cc5-8da4-b7c52c4069bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8e391c97-fd83-4e14-9a56-705ac4772eac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "7d29e3233b58d6f006cc5bf97bb6a70deb1b4af47a0b0b1c899d78ab18a71207", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "08abc25a-6a81-416c-ba60-bf19b7676cfd", "node_type": "1", "metadata": {}, "hash": "1cd4d1c1c29961e9d79825a1731ce00bded17e7695152c853719ec3a857dc5dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2980, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DeBERTa

## Overview

The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google's
BERT model released in 2018 and Facebook's RoBERTa model released in 2019.

It builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in
RoBERTa.

The abstract from the paper is the following:

*Recent progress in pre-trained neural language models has significantly improved the performance of many natural
language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with
disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the
disentangled attention mechanism, where each word is represented using two vectors that encode its content and
position, respectively, and the attention weights among words are computed using disentangled matrices on their
contents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to
predict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency
of model pretraining and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of
the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%
(90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and
pre-trained models will be made publicly available at https://github.com/microsoft/DeBERTa.*


This model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was
contributed by [kamalkraj](https://huggingface.co/kamalkraj) . The original code can be found [here](https://github.com/microsoft/DeBERTa).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DeBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on how to [Accelerate Large Model Training using DeepSpeed](https://huggingface.co/blog/accelerate-deepspeed) with DeBERTa.
- A blog post on [Supercharged Customer Service with Machine Learning](https://huggingface.co/blog/supercharge-customer-service-with-machine-learning) with DeBERTa.
- [`DebertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).


key: _node_content
string_value:
{"id_": "08abc25a-6a81-416c-ba60-bf19b7676cfd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8e391c97-fd83-4e14-9a56-705ac4772eac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "7d29e3233b58d6f006cc5bf97bb6a70deb1b4af47a0b0b1c899d78ab18a71207", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "88fb8331-2e21-4cc5-8da4-b7c52c4069bc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "0bd106b6e9e2af5a1705323ce09ffa7151cee2ee247c753feeb54066b4801c4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59bf356c-f7a8-474e-bde6-a8caa32fcef9", "node_type": "1", "metadata": {}, "hash": "af747be1a8ae039b7671897aec2447e6c53c8110750de1bea44b7840769049e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1733, "end_char_idx": 4954, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was
contributed by [kamalkraj](https://huggingface.co/kamalkraj) . The original code can be found [here](https://github.com/microsoft/DeBERTa).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DeBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on how to [Accelerate Large Model Training using DeepSpeed](https://huggingface.co/blog/accelerate-deepspeed) with DeBERTa.
- A blog post on [Supercharged Customer Service with Machine Learning](https://huggingface.co/blog/supercharge-customer-service-with-machine-learning) with DeBERTa.
- [`DebertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFDebertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification" />

- [`DebertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFDebertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Byte-Pair Encoding tokenization](https://huggingface.co/course/chapter6/5?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- [`DebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFDebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).


key: _node_content
string_value:
{"id_": "59bf356c-f7a8-474e-bde6-a8caa32fcef9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8e391c97-fd83-4e14-9a56-705ac4772eac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "7d29e3233b58d6f006cc5bf97bb6a70deb1b4af47a0b0b1c899d78ab18a71207", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "08abc25a-6a81-416c-ba60-bf19b7676cfd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "155c2aba8fb9d365a8a7bc39b33fe0cb30a2a7338b8b1e49ff4503ec40822316", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "14302f69-1e94-4be5-b3cc-cefb7f119288", "node_type": "1", "metadata": {}, "hash": "b224725574b674797b49b71f1f949e24703afcdffed3f22b70a9dfd0801359c8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3695, "end_char_idx": 5882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFDebertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Byte-Pair Encoding tokenization](https://huggingface.co/course/chapter6/5?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- [`DebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFDebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`DebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFDebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.


key: _node_content
string_value:
{"id_": "14302f69-1e94-4be5-b3cc-cefb7f119288", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8e391c97-fd83-4e14-9a56-705ac4772eac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "7d29e3233b58d6f006cc5bf97bb6a70deb1b4af47a0b0b1c899d78ab18a71207", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59bf356c-f7a8-474e-bde6-a8caa32fcef9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta", "title": "DeBERTa", "tokens": 2023, "retrieve_doc": true, "source": "transformers"}, "hash": "a1035aa0869eb831097f13f817765815f2242f2b6f6e0d0cdac9d54e81042073", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4661, "end_char_idx": 7328, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFDebertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`DebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFDebertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

## DebertaConfig

[[autodoc]] DebertaConfig

## DebertaTokenizer

[[autodoc]] DebertaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## DebertaTokenizerFast

[[autodoc]] DebertaTokenizerFast
    - build_inputs_with_special_tokens
    - create_token_type_ids_from_sequences

<frameworkcontent>
<pt>

## DebertaModel

[[autodoc]] DebertaModel
    - forward

## DebertaPreTrainedModel

[[autodoc]] DebertaPreTrainedModel

## DebertaForMaskedLM

[[autodoc]] DebertaForMaskedLM
    - forward

## DebertaForSequenceClassification

[[autodoc]] DebertaForSequenceClassification
    - forward

## DebertaForTokenClassification

[[autodoc]] DebertaForTokenClassification
    - forward

## DebertaForQuestionAnswering

[[autodoc]] DebertaForQuestionAnswering
    - forward

</pt>
<tf>

## TFDebertaModel

[[autodoc]] TFDebertaModel
    - call

## TFDebertaPreTrainedModel

[[autodoc]] TFDebertaPreTrainedModel
    - call

## TFDebertaForMaskedLM

[[autodoc]] TFDebertaForMaskedLM
    - call

## TFDebertaForSequenceClassification

[[autodoc]] TFDebertaForSequenceClassification
    - call

## TFDebertaForTokenClassification

[[autodoc]] TFDebertaForTokenClassification
    - call

## TFDebertaForQuestionAnswering

[[autodoc]] TFDebertaForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "d02e1ce4-2fe4-46a2-8b28-9c8e11cb6428", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc680113-085a-49cd-b116-d0931da2a0f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "hash": "71b8743697975ed9dd5fb1bcc4b2e8bcda63d815035f1b2f21cfdec5c1a06d12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0eb07ec0-ee64-42ab-b05e-350599ed210a", "node_type": "1", "metadata": {}, "hash": "e07996bf2a19e697395b9508e3db45c7c92be03318f6429761c16d0f6cfeca27", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3288, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Hybrid Vision Transformer (ViT Hybrid)

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The hybrid Vision Transformer (ViT) model was proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition
at Scale](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk
Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob
Uszkoreit, Neil Houlsby. It's the first paper that successfully trains a Transformer encoder on ImageNet, attaining
very good results compared to familiar convolutional architectures. ViT hybrid is a slight variant of the [plain Vision Transformer](vit),
by leveraging a convolutional backbone (specifically, [BiT](bit)) whose features are used as initial "tokens" for the Transformer.

The abstract from the paper is the following:

*While the Transformer architecture has become the de-facto standard for natural language processing tasks, its
applications to computer vision remain limited. In vision, attention is either applied in conjunction with
convolutional networks, or used to replace certain components of convolutional networks while keeping their overall
structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to
sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of
data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),
Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring
substantially fewer computational resources to train.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be
found [here](https://github.com/google-research/vision_transformer).

## Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ViTHybridForImageClassification
model = ViTHybridForImageClassification.from_pretrained("google/vit-hybrid-base-bit-384", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).


key: _node_content
string_value:
{"id_": "0eb07ec0-ee64-42ab-b05e-350599ed210a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc680113-085a-49cd-b116-d0931da2a0f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "hash": "71b8743697975ed9dd5fb1bcc4b2e8bcda63d815035f1b2f21cfdec5c1a06d12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d02e1ce4-2fe4-46a2-8b28-9c8e11cb6428", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "hash": "add417ea3da28fc723569420fb5d4167d67ff5b2c12407fd42bc7ca0c4214155", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c47d8d3-418f-4aa5-8b47-a74afc112d54", "node_type": "1", "metadata": {}, "hash": "b200e2f090e2f29b50347bd78dd1c7213575730b3b0b2c2114833452ae213b2d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1693, "end_char_idx": 5044, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When pre-trained on large amounts of
data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),
Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring
substantially fewer computational resources to train.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be
found [here](https://github.com/google-research/vision_transformer).

## Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ViTHybridForImageClassification
model = ViTHybridForImageClassification.from_pretrained("google/vit-hybrid-base-bit-384", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-hybrid-base-bit-384` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        29 |                                        18 |                      1.61 |
|            2 |                                        26 |                                        18 |                      1.44 |
|            4 |                                        25 |                                        18 |                      1.39 |
|            8 |                                        34 |                                        24 |                      1.42 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT Hybrid.

<PipelineTag pipeline="image-classification"/>

- [`ViTHybridForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "3c47d8d3-418f-4aa5-8b47-a74afc112d54", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc680113-085a-49cd-b116-d0931da2a0f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "hash": "71b8743697975ed9dd5fb1bcc4b2e8bcda63d815035f1b2f21cfdec5c1a06d12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0eb07ec0-ee64-42ab-b05e-350599ed210a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit_hybrid", "title": "Hybrid Vision Transformer (ViT Hybrid)", "tokens": 1347, "retrieve_doc": true, "source": "transformers"}, "hash": "e1d1104eea8592e6e3389ab400e3d3264a984d2273e42dd552177ee42c8ca74d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3251, "end_char_idx": 5331, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-hybrid-base-bit-384` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        29 |                                        18 |                      1.61 |
|            2 |                                        26 |                                        18 |                      1.44 |
|            4 |                                        25 |                                        18 |                      1.39 |
|            8 |                                        34 |                                        24 |                      1.42 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT Hybrid.

<PipelineTag pipeline="image-classification"/>

- [`ViTHybridForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ViTHybridConfig

[[autodoc]] ViTHybridConfig

## ViTHybridImageProcessor

[[autodoc]] ViTHybridImageProcessor
    - preprocess

## ViTHybridModel

[[autodoc]] ViTHybridModel
    - forward

## ViTHybridForImageClassification

[[autodoc]] ViTHybridForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "83973898-03b9-4127-8e5a-b4917d0b17a6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95167354-045f-4e9c-b0da-dbcaab562c55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "hash": "bf2e6f54d04f7255bf3b7bdaa3ffcbdb0ba0647108f09e6455886d0f87340d28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45f59fbb-b858-43bc-a508-3f53ae086338", "node_type": "1", "metadata": {}, "hash": "f3fda78d8cec731a0a25155df42ed80eae6a45d0874dedcb8401dfeb080669a6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3324, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Data2Vec

## Overview

The Data2Vec model was proposed in [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://arxiv.org/pdf/2202.03555) by Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu and Michael Auli.
Data2Vec proposes a unified framework for self-supervised learning across different data modalities - text, audio and images.
Importantly, predicted targets for pre-training are contextualized latent representations of the inputs, rather than modality-specific, context-independent targets.

The abstract from the paper is the following:

*While the general idea of self-supervised learning is identical across modalities, the actual algorithms and
objectives differ widely because they were developed with a single modality in mind. To get us closer to general
self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech,
NLP or computer vision. The core idea is to predict latent representations of the full input data based on a
masked view of the input in a selfdistillation setup using a standard Transformer architecture.
Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which
are local in nature, data2vec predicts contextualized latent representations that contain information from
the entire input. Experiments on the major benchmarks of speech recognition, image classification, and
natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.
Models and code are available at www.github.com/pytorch/fairseq/tree/master/examples/data2vec.*

This model was contributed by [edugp](https://huggingface.co/edugp) and [patrickvonplaten](https://huggingface.co/patrickvonplaten).
[sayakpaul](https://github.com/sayakpaul) and [Rocketknight1](https://github.com/Rocketknight1) contributed Data2Vec for vision in TensorFlow.

The original code (for NLP and Speech) can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/data2vec).
The original code for vision can be found [here](https://github.com/facebookresearch/data2vec_vision/tree/main/beit).

## Usage tips

- Data2VecAudio, Data2VecText, and Data2VecVision have all been trained using the same self-supervised learning method.
- For Data2VecAudio, preprocessing is identical to [`Wav2Vec2Model`], including feature extraction
- For Data2VecText, preprocessing is identical to [`RobertaModel`], including tokenization.
- For Data2VecVision, preprocessing is identical to [`BeitModel`], including feature extraction.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Data2Vec.

<PipelineTag pipeline="image-classification"/>

- [`Data2VecVisionForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- To fine-tune [`TFData2VecVisionForImageClassification`] on a custom dataset, see [this notebook](https://colab.research.google.com/github/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb).


key: _node_content
string_value:
{"id_": "45f59fbb-b858-43bc-a508-3f53ae086338", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95167354-045f-4e9c-b0da-dbcaab562c55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "hash": "bf2e6f54d04f7255bf3b7bdaa3ffcbdb0ba0647108f09e6455886d0f87340d28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83973898-03b9-4127-8e5a-b4917d0b17a6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "hash": "8f043e409827f462193d27570761f6412e7dad161ff33dbd73fa106276768381", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "069f6563-f3f3-4c91-b5d1-76212bc75871", "node_type": "1", "metadata": {}, "hash": "46711f5e54e329fc98770ff2f6837bd8470a8bf0867e251fa035b949cbe3826c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1819, "end_char_idx": 4319, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[sayakpaul](https://github.com/sayakpaul) and [Rocketknight1](https://github.com/Rocketknight1) contributed Data2Vec for vision in TensorFlow.

The original code (for NLP and Speech) can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/data2vec).
The original code for vision can be found [here](https://github.com/facebookresearch/data2vec_vision/tree/main/beit).

## Usage tips

- Data2VecAudio, Data2VecText, and Data2VecVision have all been trained using the same self-supervised learning method.
- For Data2VecAudio, preprocessing is identical to [`Wav2Vec2Model`], including feature extraction
- For Data2VecText, preprocessing is identical to [`RobertaModel`], including tokenization.
- For Data2VecVision, preprocessing is identical to [`BeitModel`], including feature extraction.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Data2Vec.

<PipelineTag pipeline="image-classification"/>

- [`Data2VecVisionForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- To fine-tune [`TFData2VecVisionForImageClassification`] on a custom dataset, see [this notebook](https://colab.research.google.com/github/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb).

**Data2VecText documentation resources**
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

**Data2VecAudio documentation resources**
- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

**Data2VecVision documentation resources**
- [Image classification](../tasks/image_classification)
- [Semantic segmentation](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "069f6563-f3f3-4c91-b5d1-76212bc75871", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "95167354-045f-4e9c-b0da-dbcaab562c55", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "hash": "bf2e6f54d04f7255bf3b7bdaa3ffcbdb0ba0647108f09e6455886d0f87340d28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45f59fbb-b858-43bc-a508-3f53ae086338", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/data2vec", "title": "Data2Vec", "tokens": 1654, "retrieve_doc": true, "source": "transformers"}, "hash": "153f27fdc57cc656525f3b43af8c4431b3b78265a6dc7de42d005c00bceecd4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2532, "end_char_idx": 6185, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- For Data2VecVision, preprocessing is identical to [`BeitModel`], including feature extraction.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Data2Vec.

<PipelineTag pipeline="image-classification"/>

- [`Data2VecVisionForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- To fine-tune [`TFData2VecVisionForImageClassification`] on a custom dataset, see [this notebook](https://colab.research.google.com/github/sayakpaul/TF-2.0-Hacks/blob/master/data2vec_vision_image_classification.ipynb).

**Data2VecText documentation resources**
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

**Data2VecAudio documentation resources**
- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

**Data2VecVision documentation resources**
- [Image classification](../tasks/image_classification)
- [Semantic segmentation](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## Data2VecTextConfig

[[autodoc]] Data2VecTextConfig

## Data2VecAudioConfig

[[autodoc]] Data2VecAudioConfig

## Data2VecVisionConfig

[[autodoc]] Data2VecVisionConfig

<frameworkcontent>
<pt>

## Data2VecAudioModel

[[autodoc]] Data2VecAudioModel
    - forward

## Data2VecAudioForAudioFrameClassification

[[autodoc]] Data2VecAudioForAudioFrameClassification
    - forward

## Data2VecAudioForCTC

[[autodoc]] Data2VecAudioForCTC
    - forward

## Data2VecAudioForSequenceClassification

[[autodoc]] Data2VecAudioForSequenceClassification
    - forward

## Data2VecAudioForXVector

[[autodoc]] Data2VecAudioForXVector
    - forward

## Data2VecTextModel

[[autodoc]] Data2VecTextModel
    - forward

## Data2VecTextForCausalLM

[[autodoc]] Data2VecTextForCausalLM
    - forward

## Data2VecTextForMaskedLM

[[autodoc]] Data2VecTextForMaskedLM
    - forward

## Data2VecTextForSequenceClassification

[[autodoc]] Data2VecTextForSequenceClassification
    - forward

## Data2VecTextForMultipleChoice

[[autodoc]] Data2VecTextForMultipleChoice
    - forward

## Data2VecTextForTokenClassification

[[autodoc]] Data2VecTextForTokenClassification
    - forward

## Data2VecTextForQuestionAnswering

[[autodoc]] Data2VecTextForQuestionAnswering
    - forward

## Data2VecVisionModel

[[autodoc]] Data2VecVisionModel
    - forward

## Data2VecVisionForImageClassification

[[autodoc]] Data2VecVisionForImageClassification
    - forward

## Data2VecVisionForSemanticSegmentation

[[autodoc]] Data2VecVisionForSemanticSegmentation
    - forward

</pt>
<tf>

## TFData2VecVisionModel

[[autodoc]] TFData2VecVisionModel
    - call

## TFData2VecVisionForImageClassification

[[autodoc]] TFData2VecVisionForImageClassification
    - call

## TFData2VecVisionForSemanticSegmentation

[[autodoc]] TFData2VecVisionForSemanticSegmentation
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "bf6f8d95-65c0-4a56-94a7-86b95acaabba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ul2", "title": "UL2", "tokens": 766, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bdc0017a-b3a8-4e96-b365-7305e78d13c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ul2", "title": "UL2", "tokens": 766, "retrieve_doc": true, "source": "transformers"}, "hash": "0f7423d43fd7ecc10494e86928044fb97f68d10f982c9ec431de3e832b804941", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# UL2

## Overview

The T5 model was presented in [Unifying Language Learning Paradigms](https://arxiv.org/pdf/2205.05131v1.pdf) by Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald Metzler.

The abstract from the paper is the following:

*Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 and/or GPT-like models across multiple diverse setups. Finally, by scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised NLP tasks ranging from language generation (with automated and human evaluation), language understanding, text classification, question answering, commonsense reasoning, long text reasoning, structured knowledge grounding and information retrieval. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.*

This model was contributed by [DanielHesslow](https://huggingface.co/Seledorn). The original code can be found [here](https://github.com/google-research/google-research/tree/master/ul2).

## Usage tips

- UL2 is an encoder-decoder model pre-trained on a mixture of denoising functions as well as fine-tuned on an array of downstream tasks.
- UL2 has the same architecture as [T5v1.1](t5v1.1) but uses the Gated-SiLU activation function instead of Gated-GELU.
- The authors release checkpoints of one architecture which can be seen [here](https://huggingface.co/google/ul2)

<Tip> 

As UL2 has the same architecture as T5v1.1,  refer to [T5's documentation page](t5) for API reference, tips, code examples and notebooks.

</Tip>


key: _node_content
string_value:
{"id_": "47d23938-5d34-4849-bb06-5f3b29da0bde", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/olmo", "title": "OLMo", "tokens": 782, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "83c3daf7-9492-41cd-8b53-050ecd7513fb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/olmo", "title": "OLMo", "tokens": 782, "retrieve_doc": true, "source": "transformers"}, "hash": "c24d38db1fbfbff20ee8bdf6009580154a1ae31e9ce8ddc37dee78cee89a5c34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2473, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OLMo

## Overview

The OLMo model was proposed in [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838) by Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi.

OLMo is a series of **O**pen **L**anguage **Mo**dels designed to enable the science of language models. The OLMo models are trained on the Dolma dataset. We release all code, checkpoints, logs (coming soon), and details involved in training these models.

The abstract from the paper is the following:

*Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, this technical report details the first release of OLMo, a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.*

This model was contributed by [shanearora](https://huggingface.co/shanearora).
The original code can be found [here](https://github.com/allenai/OLMo/tree/main/olmo).


## OlmoConfig

[[autodoc]] OlmoConfig

## OlmoModel

[[autodoc]] OlmoModel
    - forward

## OlmoForCausalLM

[[autodoc]] OlmoForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "1d9e55e2-ba2c-45c5-a41f-b16ddc6958bb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d5790776-f8ff-41c0-bd37-65e0f855dd24", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04487e34-bcef-4a32-beee-77c762557d4d", "node_type": "1", "metadata": {}, "hash": "99e106084bac918565a3c12c00bc69a426deb4a3c463552a4a4b211adde43dc8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2887, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLM-RoBERTa

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=xlm-roberta">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlm--roberta-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/xlm-roberta-base">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The XLM-RoBERTa model was proposed in [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume
Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. It is based on Facebook's
RoBERTa model released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl
data.

The abstract from the paper is the following:

*This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a
wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred
languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly
outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on
XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on
low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We
also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the
trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource
languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing
per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We
will make XLM-R code, data, and models publicly available.*

This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).

## Usage tips

- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does
  not require `lang` tensors to understand which language is used, and should be able to determine the correct
  language from the input ids.
- Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses masked language modeling on sentences coming from one language.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with XLM-RoBERTa.


key: _node_content
string_value:
{"id_": "04487e34-bcef-4a32-beee-77c762557d4d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d5790776-f8ff-41c0-bd37-65e0f855dd24", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d9e55e2-ba2c-45c5-a41f-b16ddc6958bb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "615be13e47ff6572bf1934b05287ebd8ae6813b65c53ab05aab268101c6505d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee4c5f59-50af-4fd1-8543-a565b782ff49", "node_type": "1", "metadata": {}, "hash": "05d9756023dcb8700a311a9568141a8b78c722816122eb1d5f8c2bcb55ece4be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1436, "end_char_idx": 4777, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
XLM-R performs particularly well on
low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We
also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the
trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource
languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing
per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We
will make XLM-R code, data, and models publicly available.*

This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).

## Usage tips

- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does
  not require `lang` tensors to understand which language is used, and should be able to determine the correct
  language from the input ids.
- Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses masked language modeling on sentences coming from one language.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with XLM-RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on how to [finetune XLM RoBERTa for multiclass classification with Habana Gaudi on AWS](https://www.philschmid.de/habana-distributed-training)
- [`XLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification](https://huggingface.co/docs/transformers/tasks/sequence_classification) chapter of the 🤗 Hugging Face Task Guides.
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`XLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).


key: _node_content
string_value:
{"id_": "ee4c5f59-50af-4fd1-8543-a565b782ff49", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d5790776-f8ff-41c0-bd37-65e0f855dd24", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04487e34-bcef-4a32-beee-77c762557d4d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "2add8e5686903b3ade58647c94ba142b7737fbcc3ea8366ddd57cfba7595b8bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bf16f1fb-8208-410f-b6c2-a61d35d58c40", "node_type": "1", "metadata": {}, "hash": "3dd21a269c51f98ea7d3ae33c38a42845f150db3e8dd08c84ada06a61bbb894c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3016, "end_char_idx": 6328, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on how to [finetune XLM RoBERTa for multiclass classification with Habana Gaudi on AWS](https://www.philschmid.de/habana-distributed-training)
- [`XLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification](https://huggingface.co/docs/transformers/tasks/sequence_classification) chapter of the 🤗 Hugging Face Task Guides.
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`XLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="text-generation"/>

- [`XLMRobertaForCausalLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) chapter of the 🤗 Hugging Face Task Guides.
- [Causal language modeling task guide](../tasks/language_modeling)

<PipelineTag pipeline="fill-mask"/>

- [`XLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).


key: _node_content
string_value:
{"id_": "bf16f1fb-8208-410f-b6c2-a61d35d58c40", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d5790776-f8ff-41c0-bd37-65e0f855dd24", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee4c5f59-50af-4fd1-8543-a565b782ff49", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "41fb83122d2b71c7ad018ab8014a80c8c533d6d41edfec5d26be4b5897a4669d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a90b80b-6757-4ede-9345-d83e6f8e2b01", "node_type": "1", "metadata": {}, "hash": "7ac9b404f8f972cd197f72bcc8325504c1a132e5b041419085aecc7b3a5bca1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4778, "end_char_idx": 7924, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="text-generation"/>

- [`XLMRobertaForCausalLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) chapter of the 🤗 Hugging Face Task Guides.
- [Causal language modeling task guide](../tasks/language_modeling)

<PipelineTag pipeline="fill-mask"/>

- [`XLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`XLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).


key: _node_content
string_value:
{"id_": "5a90b80b-6757-4ede-9345-d83e6f8e2b01", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d5790776-f8ff-41c0-bd37-65e0f855dd24", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf16f1fb-8208-410f-b6c2-a61d35d58c40", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "273b518f4fa0057023fd479fbf1d8f3a94f3b40a39bf18761ff3305b1d1259ed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "262e9240-7fac-4e3f-9dc4-daa33e6fcee8", "node_type": "1", "metadata": {}, "hash": "abbe03ec8c65de72df62e0564b7606f3650996465bbe73078598c3a41a1ed85f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6626, "end_char_idx": 9091, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`XLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**

- [`XLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFXLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

🚀 Deploy

- A blog post on how to [Deploy Serverless XLM RoBERTa on AWS Lambda](https://www.philschmid.de/multilingual-serverless-xlm-roberta-with-huggingface).

<Tip> 

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well as the information relative to the inputs and outputs.


key: _node_content
string_value:
{"id_": "262e9240-7fac-4e3f-9dc4-daa33e6fcee8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d5790776-f8ff-41c0-bd37-65e0f855dd24", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a90b80b-6757-4ede-9345-d83e6f8e2b01", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "7a594156247f2fa2d9fc9d8e8eca831386d0d3d9f00e3dcce6fa001640c9ba5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7757, "end_char_idx": 11332, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**

- [`XLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFXLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

🚀 Deploy

- A blog post on how to [Deploy Serverless XLM RoBERTa on AWS Lambda](https://www.philschmid.de/multilingual-serverless-xlm-roberta-with-huggingface).

<Tip> 

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well as the information relative to the inputs and outputs.
</Tip>

## XLMRobertaConfig

[[autodoc]] XLMRobertaConfig

## XLMRobertaTokenizer

[[autodoc]] XLMRobertaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## XLMRobertaTokenizerFast

[[autodoc]] XLMRobertaTokenizerFast

<frameworkcontent>
<pt>

## XLMRobertaModel

[[autodoc]] XLMRobertaModel
    - forward

## XLMRobertaForCausalLM

[[autodoc]] XLMRobertaForCausalLM
    - forward

## XLMRobertaForMaskedLM

[[autodoc]] XLMRobertaForMaskedLM
    - forward

## XLMRobertaForSequenceClassification

[[autodoc]] XLMRobertaForSequenceClassification
    - forward

## XLMRobertaForMultipleChoice

[[autodoc]] XLMRobertaForMultipleChoice
    - forward

## XLMRobertaForTokenClassification

[[autodoc]] XLMRobertaForTokenClassification
    - forward

## XLMRobertaForQuestionAnswering

[[autodoc]] XLMRobertaForQuestionAnswering
    - forward

</pt>
<tf>

## TFXLMRobertaModel

[[autodoc]] TFXLMRobertaModel
    - call

## TFXLMRobertaForCausalLM

[[autodoc]] TFXLMRobertaForCausalLM
    - call

## TFXLMRobertaForMaskedLM

[[autodoc]] TFXLMRobertaForMaskedLM
    - call

## TFXLMRobertaForSequenceClassification

[[autodoc]] TFXLMRobertaForSequenceClassification
    - call

## TFXLMRobertaForMultipleChoice

[[autodoc]] TFXLMRobertaForMultipleChoice
    - call

## TFXLMRobertaForTokenClassification

[[autodoc]] TFXLMRobertaForTokenClassification
    - call

## TFXLMRobertaForQuestionAnswering

[[autodoc]] TFXLMRobertaForQuestionAnswering
    - call

</tf>
<jax>

## FlaxXLMRobertaModel

[[autodoc]] FlaxXLMRobertaModel
    - __call__

## FlaxXLMRobertaForCausalLM

[[autodoc]] FlaxXLMRobertaForCausalLM
    - __call__

## FlaxXLMRobertaForMaskedLM

[[autodoc]] FlaxXLMRobertaForMaskedLM
    - __call__

## FlaxXLMRobertaForSequenceClassification

[[autodoc]] FlaxXLMRobertaForSequenceClassification
    - __call__

## FlaxXLMRobertaForMultipleChoice

[[autodoc]] FlaxXLMRobertaForMultipleChoice
    - __call__

## FlaxXLMRobertaForTokenClassification

[[autodoc]] FlaxXLMRobertaForTokenClassification
    - __call__

## FlaxXLMRobertaForQuestionAnswering

[[autodoc]] FlaxXLMRobertaForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "1d96fae0-1ada-4030-ad00-afdb091b64c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eaeab45-bdc7-4dc9-a994-1911f788e852", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "f2d57915678ec31d2e39d54ce890a22db5faf6571ff26d3ab437a77dd1829631", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "894ef7c9-fdea-4c35-a8b1-06b21e626a46", "node_type": "1", "metadata": {}, "hash": "876660f27fc4b678cb5a0b4b1901bc313f02d3a85b06db95b633449bd9ef95fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2671, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DiT

## Overview

DiT was proposed in [DiT: Self-supervised Pre-training for Document Image Transformer](https://arxiv.org/abs/2203.02378) by Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei.
DiT applies the self-supervised objective of [BEiT](beit) (BERT pre-training of Image Transformers) to 42 million document images, allowing for state-of-the-art results on tasks including:

- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of
  400,000 images belonging to one of 16 classes).
- document layout analysis: the [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) dataset (a collection of more
  than 360,000 document images constructed by automatically parsing PubMed XML files).
- table detection: the [ICDAR 2019 cTDaR](https://github.com/cndplab-founder/ICDAR2019_cTDaR) dataset (a collection of
  600 training images and 240 testing images).

The abstract from the paper is the following:

*Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 → 92.69), document layout analysis (91.0 → 94.9) and table detection (94.23 → 96.55). *

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dit_architecture.jpg"
alt="drawing" width="600"/> 

<small> Summary of the approach. Taken from the [original paper](https://arxiv.org/abs/2203.02378). </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/dit).

## Usage tips

One can directly use the weights of DiT with the AutoModel API:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/dit-base")
```

This will load the model pre-trained on masked image modeling. Note that this won't include the language modeling head on top, used to predict visual tokens.


key: _node_content
string_value:
{"id_": "894ef7c9-fdea-4c35-a8b1-06b21e626a46", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eaeab45-bdc7-4dc9-a994-1911f788e852", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "f2d57915678ec31d2e39d54ce890a22db5faf6571ff26d3ab437a77dd1829631", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d96fae0-1ada-4030-ad00-afdb091b64c3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "80f383e52accb56508999ad93454a1aa07c9c822dedef86b7c2056380fced41e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a78e972-a680-4e21-b130-a09d66d170dc", "node_type": "1", "metadata": {}, "hash": "8c6628c7afc940abd54804257dbf62455498a0e0ee6e9b3119b0c8272d6f26ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 936, "end_char_idx": 4283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract from the paper is the following:

*Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 → 92.69), document layout analysis (91.0 → 94.9) and table detection (94.23 → 96.55). *

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dit_architecture.jpg"
alt="drawing" width="600"/> 

<small> Summary of the approach. Taken from the [original paper](https://arxiv.org/abs/2203.02378). </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/dit).

## Usage tips

One can directly use the weights of DiT with the AutoModel API:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/dit-base")
```

This will load the model pre-trained on masked image modeling. Note that this won't include the language modeling head on top, used to predict visual tokens.

To include the head, you can load the weights into a `BeitForMaskedImageModeling` model, like so:

```python
from transformers import BeitForMaskedImageModeling

model = BeitForMaskedImageModeling.from_pretrained("microsoft/dit-base")
```

You can also load a fine-tuned model from the [hub](https://huggingface.co/models?other=dit), like so:

```python
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained("microsoft/dit-base-finetuned-rvlcdip")
```

This particular checkpoint was fine-tuned on [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/), an important benchmark for document image classification.
A notebook that illustrates inference for document image classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DiT/Inference_with_DiT_(Document_Image_Transformer)_for_document_image_classification.ipynb).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DiT.

<PipelineTag pipeline="image-classification"/>

- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "7a78e972-a680-4e21-b130-a09d66d170dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8eaeab45-bdc7-4dc9-a994-1911f788e852", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "f2d57915678ec31d2e39d54ce890a22db5faf6571ff26d3ab437a77dd1829631", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "894ef7c9-fdea-4c35-a8b1-06b21e626a46", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "33bc599b9ecf99b5be6fe3ccee1eaef38d0eacb86a4ec42b7c6f1baaaaaedc11", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2577, "end_char_idx": 4447, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that this won't include the language modeling head on top, used to predict visual tokens.

To include the head, you can load the weights into a `BeitForMaskedImageModeling` model, like so:

```python
from transformers import BeitForMaskedImageModeling

model = BeitForMaskedImageModeling.from_pretrained("microsoft/dit-base")
```

You can also load a fine-tuned model from the [hub](https://huggingface.co/models?other=dit), like so:

```python
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained("microsoft/dit-base-finetuned-rvlcdip")
```

This particular checkpoint was fine-tuned on [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/), an important benchmark for document image classification.
A notebook that illustrates inference for document image classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DiT/Inference_with_DiT_(Document_Image_Transformer)_for_document_image_classification.ipynb).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DiT.

<PipelineTag pipeline="image-classification"/>

- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<Tip>

  As DiT's architecture is equivalent to that of BEiT, one can refer to [BEiT's documentation page](beit) for all tips, code examples and notebooks.
</Tip>


key: _node_content
string_value:
{"id_": "e5ebcacc-9a6a-41f1-874a-9a1f4087fdfd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2fca0ddb-e7be-4f97-929d-3e6ec95c31da", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "hash": "82abd746c8212e2ff0ead89b54433298e514e295d5da65f48dee7dd9605abe97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10ad0897-2bbe-4fec-8b4b-59d4706d475b", "node_type": "1", "metadata": {}, "hash": "da9fa83b82e1fb342718f431af64877b257802249ba234567691f7a88565c418", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3356, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TVLT

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The TVLT model was proposed in [TVLT: Textless Vision-Language Transformer](https://arxiv.org/abs/2209.14156)
by Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal (the first three authors contributed equally). The Textless Vision-Language Transformer (TVLT) is a model that uses raw visual and audio inputs for vision-and-language representation learning, without using text-specific modules such as tokenization or automatic speech recognition (ASR). It can perform various audiovisual and vision-language tasks like retrieval, question answering, etc.

The abstract from the paper is the following:

*In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text.*

<p align="center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvlt_architecture.png"
alt="drawing" width="600"/>
</p>

<small> TVLT architecture. Taken from the <a href="[https://arxiv.org/abs/2102.03334](https://arxiv.org/abs/2209.14156)">original paper</a>. </small>

The original code can be found [here](https://github.com/zinengtang/TVLT). This model was contributed by [Zineng Tang](https://huggingface.co/ZinengTang).

## Usage tips

- TVLT is a model that takes both `pixel_values` and `audio_values` as input. One can use [`TvltProcessor`] to prepare data for the model.
  This processor wraps an image processor (for the image/video modality) and an audio feature extractor (for the audio modality) into one.
- TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a `pixel_mask` that indicates which pixels are real/padding and `audio_mask` that indicates which audio values are real/padding.
- The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in [ViTMAE](vitmae). The difference is that the model includes embedding layers for the audio modality.
- The PyTorch version of this model is only available in torch 1.10 and higher.


key: _node_content
string_value:
{"id_": "10ad0897-2bbe-4fec-8b4b-59d4706d475b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2fca0ddb-e7be-4f97-929d-3e6ec95c31da", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "hash": "82abd746c8212e2ff0ead89b54433298e514e295d5da65f48dee7dd9605abe97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5ebcacc-9a6a-41f1-874a-9a1f4087fdfd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "hash": "5d7682962b4a1005e76baa9ae84911e048d7db14bbd7e43fd6da3502194555bb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1898, "end_char_idx": 3818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

<p align="center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvlt_architecture.png"
alt="drawing" width="600"/>
</p>

<small> TVLT architecture. Taken from the <a href="[https://arxiv.org/abs/2102.03334](https://arxiv.org/abs/2209.14156)">original paper</a>. </small>

The original code can be found [here](https://github.com/zinengtang/TVLT). This model was contributed by [Zineng Tang](https://huggingface.co/ZinengTang).

## Usage tips

- TVLT is a model that takes both `pixel_values` and `audio_values` as input. One can use [`TvltProcessor`] to prepare data for the model.
  This processor wraps an image processor (for the image/video modality) and an audio feature extractor (for the audio modality) into one.
- TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a `pixel_mask` that indicates which pixels are real/padding and `audio_mask` that indicates which audio values are real/padding.
- The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in [ViTMAE](vitmae). The difference is that the model includes embedding layers for the audio modality.
- The PyTorch version of this model is only available in torch 1.10 and higher.

## TvltConfig

[[autodoc]] TvltConfig

## TvltProcessor

[[autodoc]] TvltProcessor
    - __call__

## TvltImageProcessor

[[autodoc]] TvltImageProcessor
    - preprocess

## TvltFeatureExtractor

[[autodoc]] TvltFeatureExtractor
    - __call__

## TvltModel

[[autodoc]] TvltModel
    - forward

## TvltForPreTraining

[[autodoc]] TvltForPreTraining
    - forward

## TvltForAudioVisualClassification

[[autodoc]] TvltForAudioVisualClassification
    - forward


key: _node_content
string_value:
{"id_": "afa98d94-89a9-45e3-8676-ca7a05547f55", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e61fc782-82c3-46f8-a53b-3f4c656148ea", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "hash": "7cae33e702575ee5a646b9f33883830e70f6c2e50d4bc45131937ccc0cd24916", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "710b5cfb-b4b1-4a39-a34d-f6b5a760ee4d", "node_type": "1", "metadata": {}, "hash": "8494db7d7bea4a45969be730d77aa597aea130a6b8af4b027529345a48f9b07d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Time Series Transformer

## Overview

The Time Series Transformer model is a vanilla encoder-decoder Transformer for time series forecasting.
This model was contributed by [kashif](https://huggingface.co/kashif).

## Usage tips

- Similar to other models in the library, [`TimeSeriesTransformerModel`] is the raw Transformer without any head on top, and [`TimeSeriesTransformerForPrediction`]
adds a distribution head on top of the former, which can be used for time-series forecasting. Note that this is a so-called probabilistic forecasting model, not a
point forecasting model. This means that the model learns a distribution, from which one can sample. The model doesn't directly output values.
- [`TimeSeriesTransformerForPrediction`] consists of 2 blocks: an encoder, which takes a `context_length` of time series values as input (called `past_values`),
and a decoder, which predicts a `prediction_length` of time series values into the future (called `future_values`). During training, one needs to provide
pairs of (`past_values` and `future_values`) to the model.
- In addition to the raw (`past_values` and `future_values`), one typically provides additional features to the model. These can be the following:
    - `past_time_features`: temporal features which the model will add to `past_values`. These serve as "positional encodings" for the Transformer encoder.
    Examples are "day of the month", "month of the year", etc. as scalar values (and then stacked together as a vector).
    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being "day of the month", 8 being "month of the year").
    - `future_time_features`: temporal features which the model will add to `future_values`. These serve as "positional encodings" for the Transformer decoder.
    Examples are "day of the month", "month of the year", etc. as scalar values (and then stacked together as a vector).
    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being "day of the month", 8 being "month of the year").
    - `static_categorical_features`: categorical features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the store ID or region ID that identifies a given time-series.
    Note that these features need to be known for ALL data points (also those in the future).
    - `static_real_features`: real-valued features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the image representation of the product for which you have the time-series values (like the [ResNet](resnet) embedding of a "shoe" picture,
    if your time-series is about the sales of shoes).
    Note that these features need to be known for ALL data points (also those in the future).
- The model is trained using "teacher-forcing", similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the
`future_values` one position to the right as input to the decoder, prepended by the last value of `past_values`. At each time step, the model needs to predict the
next target.


key: _node_content
string_value:
{"id_": "710b5cfb-b4b1-4a39-a34d-f6b5a760ee4d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e61fc782-82c3-46f8-a53b-3f4c656148ea", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "hash": "7cae33e702575ee5a646b9f33883830e70f6c2e50d4bc45131937ccc0cd24916", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "afa98d94-89a9-45e3-8676-ca7a05547f55", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "hash": "363d596799887eef9b3e9dc54b9aac6d6ffda92a26007a1dabef3f2843ea3dd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1688, "end_char_idx": 4552, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `future_time_features`: temporal features which the model will add to `future_values`. These serve as "positional encodings" for the Transformer decoder.
    Examples are "day of the month", "month of the year", etc. as scalar values (and then stacked together as a vector).
    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being "day of the month", 8 being "month of the year").
    - `static_categorical_features`: categorical features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the store ID or region ID that identifies a given time-series.
    Note that these features need to be known for ALL data points (also those in the future).
    - `static_real_features`: real-valued features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the image representation of the product for which you have the time-series values (like the [ResNet](resnet) embedding of a "shoe" picture,
    if your time-series is about the sales of shoes).
    Note that these features need to be known for ALL data points (also those in the future).
- The model is trained using "teacher-forcing", similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the
`future_values` one position to the right as input to the decoder, prepended by the last value of `past_values`. At each time step, the model needs to predict the
next target. So the set-up of training is similar to a GPT model for language, except that there's no notion of `decoder_start_token_id` (we just use the last value
of the context as initial input for the decoder).
- At inference time, we give the final value of the `past_values` as input to the decoder. Next, we can sample from the model to make a prediction at the next time step,
which is then fed to the decoder in order to make the next prediction (also called autoregressive generation).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- Check out the Time Series Transformer blog-post in HuggingFace blog: [Probabilistic Time Series Forecasting with 🤗 Transformers](https://huggingface.co/blog/time-series-transformers)


## TimeSeriesTransformerConfig

[[autodoc]] TimeSeriesTransformerConfig

## TimeSeriesTransformerModel

[[autodoc]] TimeSeriesTransformerModel
    - forward

## TimeSeriesTransformerForPrediction

[[autodoc]] TimeSeriesTransformerForPrediction
    - forward


key: _node_content
string_value:
{"id_": "d5fb8905-d73c-4969-bd21-ffd27bc7f483", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "085b0b7c-171c-4cc6-90ab-810092e09179", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "hash": "eb48b9f8642f3f4ce17193c44211ba0f035de510b36aef07282c1a34a26eb9fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42632243-5cca-49f2-8fd2-b87cd76244b9", "node_type": "1", "metadata": {}, "hash": "be364c94058186bd36c8af94c1aae17905e36bb9ddd49274a1afb06915b5065d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPT Neo

## Overview

The GPTNeo model was released in the [EleutherAI/gpt-neo](https://github.com/EleutherAI/gpt-neo) repository by Sid
Black, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy. It is a GPT2 like causal language model trained on the
[Pile](https://pile.eleuther.ai/) dataset.

The architecture is similar to GPT2 except that GPT Neo uses local attention in every other layer with a window size of
256 tokens.

This model was contributed by [valhalla](https://huggingface.co/valhalla).

## Usage example

The `generate()` method can be used to generate text using GPT Neo model.

```python
>>> from transformers import GPTNeoForCausalLM, GPT2Tokenizer

>>> model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")
>>> tokenizer = GPT2Tokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")

>>> prompt = (
...     "In a shocking finding, scientists discovered a herd of unicorns living in a remote, "
...     "previously unexplored valley, in the Andes Mountains. Even more surprising to the "
...     "researchers was the fact that the unicorns spoke perfect English."
... )

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

## Combining GPT-Neo and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature, and make sure your hardware is compatible with Flash-Attention 2. More details are available [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) concerning the installation.

Make sure as well to load your model in half-precision (e.g. `torch.float16`).

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"def hello_world():\n    >>> run_script("hello.py")\n    >>> exit(0)\n<|endoftext|>"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `EleutherAI/gpt-neo-2.7B` checkpoint and the Flash Attention 2 version of the model.


key: _node_content
string_value:
{"id_": "42632243-5cca-49f2-8fd2-b87cd76244b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "085b0b7c-171c-4cc6-90ab-810092e09179", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "hash": "eb48b9f8642f3f4ce17193c44211ba0f035de510b36aef07282c1a34a26eb9fb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5fb8905-d73c-4969-bd21-ffd27bc7f483", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "hash": "de04593e5ecf99ce44ff22fcc0d5ef6e716355fe275e105200c6cc08e565af49", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1589, "end_char_idx": 4065, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
More details are available [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) concerning the installation.

Make sure as well to load your model in half-precision (e.g. `torch.float16`).

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"def hello_world():\n    >>> run_script("hello.py")\n    >>> exit(0)\n<|endoftext|>"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `EleutherAI/gpt-neo-2.7B` checkpoint and the Flash Attention 2 version of the model.
Note that for GPT-Neo it is not possible to train / run on very long context as the max [position embeddings](https://huggingface.co/EleutherAI/gpt-neo-2.7B/blob/main/config.json#L58 ) is limited to 2048 - but this is applicable to all gpt-neo models and not specific to FA-2

<div style="text-align: center">
<img src="https://user-images.githubusercontent.com/49240599/272241893-b1c66b75-3a48-4265-bc47-688448568b3d.png">
</div>


## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## GPTNeoConfig

[[autodoc]] GPTNeoConfig


<frameworkcontent>
<pt>

## GPTNeoModel

[[autodoc]] GPTNeoModel
    - forward

## GPTNeoForCausalLM

[[autodoc]] GPTNeoForCausalLM
    - forward

## GPTNeoForQuestionAnswering

[[autodoc]] GPTNeoForQuestionAnswering
    - forward

## GPTNeoForSequenceClassification

[[autodoc]] GPTNeoForSequenceClassification
    - forward

## GPTNeoForTokenClassification

[[autodoc]] GPTNeoForTokenClassification
    - forward

</pt>
<jax>

## FlaxGPTNeoModel

[[autodoc]] FlaxGPTNeoModel
    - __call__

## FlaxGPTNeoForCausalLM

[[autodoc]] FlaxGPTNeoForCausalLM
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8e0b4ed2-c6a1-4e1c-b673-e7aed31bc426", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1b33534e-687e-4456-97f9-e0d3560bcbc1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "hash": "a6bad01a61ff44ec9ae62903ef441c2e7ebc1766107ca17dd47b0f07737189c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "62c451c5-90e7-4901-805e-5cb65946c8f1", "node_type": "1", "metadata": {}, "hash": "98c38ade1627c5ba36db2365fbeed9b914d7ddc61f9d73fc2cc204a14afa749b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2064, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Hubert

## Overview

Hubert was proposed in [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447) by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan
Salakhutdinov, Abdelrahman Mohamed.

The abstract from the paper is the following:

*Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are
multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training
phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we
propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an
offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our
approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined
acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised
clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means
teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the
state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h,
10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER
reduction on the more challenging dev-other and test-other evaluation subsets.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

# Usage tips

- Hubert is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- Hubert model was fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].


key: _node_content
string_value:
{"id_": "62c451c5-90e7-4901-805e-5cb65946c8f1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1b33534e-687e-4456-97f9-e0d3560bcbc1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "hash": "a6bad01a61ff44ec9ae62903ef441c2e7ebc1766107ca17dd47b0f07737189c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e0b4ed2-c6a1-4e1c-b673-e7aed31bc426", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "hash": "91c088db2258fc27afbdaf83b625efdaf3974e8e15c9a51b819829a9d5cb87a7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2067, "end_char_idx": 4488, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Using Flash Attention 2

Flash Attention 2 is an faster, optimized version of the model.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

Below is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of `facebook/hubert-large-ls960-ft`, the flash-attention-2 and the sdpa (scale-dot-product-attention) version. We show the average speedup obtained on the `librispeech_asr` `clean` validation split: 

```python
>>> from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("facebook/hubert-large-ls960-ft", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
...
```

### Expected speedups

Below is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/hubert-large-ls960-ft` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: 


<div style="text-align: center">
<img src="https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Hubert_speedup.png">
</div>


## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## HubertConfig

[[autodoc]] HubertConfig

<frameworkcontent>
<pt>

## HubertModel

[[autodoc]] HubertModel
    - forward

## HubertForCTC

[[autodoc]] HubertForCTC
    - forward

## HubertForSequenceClassification

[[autodoc]] HubertForSequenceClassification
    - forward

</pt>
<tf>

## TFHubertModel

[[autodoc]] TFHubertModel
    - call

## TFHubertForCTC

[[autodoc]] TFHubertForCTC
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "9d306cd1-d3f2-4067-974f-96948095b184", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2", "title": "Qwen2", "tokens": 795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd53def6-83e8-4428-8f1f-5744f4a358bd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2", "title": "Qwen2", "tokens": 795, "retrieve_doc": true, "source": "transformers"}, "hash": "561d8754208f82f03566c4e3f33cca0b9d34cc8f2e0dbb509b040941e8a9e133", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2374, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Qwen2

## Overview

Qwen2 is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, Qwen2-72B, Qwen2-Audio, etc.

### Model Details

Qwen2 is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.


## Usage tips

`Qwen2-7B` and `Qwen2-7B-Instruct` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)

In the following, we demonstrate how to use `Qwen2-7B-Instruct` for the inference. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-7B-Instruct", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B-Instruct")

>>> prompt = "Give me a short introduction to large language model."

>>> messages = [{"role": "user", "content": prompt}]

>>> text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

>>> model_inputs = tokenizer([text], return_tensors="pt").to(device)

>>> generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)

>>> generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]

>>> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

## Qwen2Config

[[autodoc]] Qwen2Config

## Qwen2Tokenizer

[[autodoc]] Qwen2Tokenizer
    - save_vocabulary

## Qwen2TokenizerFast

[[autodoc]] Qwen2TokenizerFast

## Qwen2Model

[[autodoc]] Qwen2Model
    - forward

## Qwen2ForCausalLM

[[autodoc]] Qwen2ForCausalLM
    - forward

## Qwen2ForSequenceClassification

[[autodoc]] Qwen2ForSequenceClassification
    - forward

## Qwen2ForTokenClassification

[[autodoc]] Qwen2ForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "5556ad70-08e0-4610-932f-12acc801e773", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41f2ab40-9708-47ac-8715-e1b5da69967e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "176d7f5ae3c2372dde1cfff67a480ebc9f911de0003830c44caf5e374da6fbf1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bec37be-6864-432c-ac68-18a855e4779a", "node_type": "1", "metadata": {}, "hash": "df4e7cbd0e87004a0f349e98dbbafbfff98526700d153be245c358a6a08a2f21", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ZoeDepth

## Overview

The ZoeDepth model was proposed in [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://arxiv.org/abs/2302.12288) by Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller. ZoeDepth extends the [DPT](dpt) framework for metric (also called absolute) depth estimation. ZoeDepth is pre-trained on 12 datasets using relative depth and fine-tuned on two domains (NYU and KITTI) using metric depth. A lightweight head is used with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier.

The abstract from the paper is the following:

*This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png"
alt="drawing" width="600"/>

<small> ZoeDepth architecture. Taken from the <a href="https://arxiv.org/abs/2302.12288">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/isl-org/ZoeDepth).

## Usage tips

- ZoeDepth is an absolute (also called metric) depth estimation model, unlike DPT which is a relative depth estimation model. This means that ZoeDepth is able to estimate depth in metric units like meters.


key: _node_content
string_value:
{"id_": "4bec37be-6864-432c-ac68-18a855e4779a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41f2ab40-9708-47ac-8715-e1b5da69967e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "176d7f5ae3c2372dde1cfff67a480ebc9f911de0003830c44caf5e374da6fbf1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5556ad70-08e0-4610-932f-12acc801e773", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "7117d21dbd2254749fde8eee9e82a610124d77e49ebea2a2afedb80c9903335c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "14423383-33a0-4083-ac84-cf76228906b8", "node_type": "1", "metadata": {}, "hash": "8a8a8d0a1620e5cab6b9d0fb8a5e4dbc071d483f8a97b8424f5332e689fe9ada", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1147, "end_char_idx": 4454, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png"
alt="drawing" width="600"/>

<small> ZoeDepth architecture. Taken from the <a href="https://arxiv.org/abs/2302.12288">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/isl-org/ZoeDepth).

## Usage tips

- ZoeDepth is an absolute (also called metric) depth estimation model, unlike DPT which is a relative depth estimation model. This means that ZoeDepth is able to estimate depth in metric units like meters.

The easiest to perform inference with ZoeDepth is by leveraging the [pipeline API](../main_classes/pipelines.md):

```python
from transformers import pipeline
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

pipe = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")
result = pipe(image)
depth = result["depth"]
```

Alternatively, one can also perform inference using the classes:

```python
from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation
import torch
import numpy as np
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti")
model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti")

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)

# visualize the prediction
output = prediction.squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(formatted)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ZoeDepth.


key: _node_content
string_value:
{"id_": "14423383-33a0-4083-ac84-cf76228906b8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "41f2ab40-9708-47ac-8715-e1b5da69967e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "176d7f5ae3c2372dde1cfff67a480ebc9f911de0003830c44caf5e374da6fbf1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4bec37be-6864-432c-ac68-18a855e4779a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "895f2a8abe348305b907b3ee3568bff408cc2ebec59c0dd2552502fb1ef0a406", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2841, "end_char_idx": 4824, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The easiest to perform inference with ZoeDepth is by leveraging the [pipeline API](../main_classes/pipelines.md):

```python
from transformers import pipeline
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

pipe = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")
result = pipe(image)
depth = result["depth"]
```

Alternatively, one can also perform inference using the classes:

```python
from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation
import torch
import numpy as np
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti")
model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti")

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)

# visualize the prediction
output = prediction.squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(formatted)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ZoeDepth.

- A demo notebook regarding inference with ZoeDepth models can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ZoeDepth). 🌎

## ZoeDepthConfig

[[autodoc]] ZoeDepthConfig

## ZoeDepthImageProcessor

[[autodoc]] ZoeDepthImageProcessor
    - preprocess

## ZoeDepthForDepthEstimation

[[autodoc]] ZoeDepthForDepthEstimation
    - forward


key: _node_content
string_value:
{"id_": "f4f3893d-8ead-4293-8588-67d0d7c3c235", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef8b91f4-bc6a-4711-adc7-07b4c541a650", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd04cbee546fc1ffa9670497c6a1c019aca937c65cfdec8c905d64deb20f251", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bd69daf8-86df-4163-97d0-01ecb98410a1", "node_type": "1", "metadata": {}, "hash": "23cfd99685570d19c96c88ce554adc2b21fc5d0abab896e08d10eb4d6cdd35d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2823, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LayoutLM

<a id='Overview'></a>

## Overview

The LayoutLM model was proposed in the paper [LayoutLM: Pre-training of Text and Layout for Document Image
Understanding](https://arxiv.org/abs/1912.13318) by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and
Ming Zhou. It's a simple but effective pretraining method of text and layout for document image understanding and
information extraction tasks, such as form understanding and receipt understanding. It obtains state-of-the-art results
on several downstream tasks:

- form understanding: the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (a collection of 199 annotated
  forms comprising more than 30,000 words).
- receipt understanding: the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset (a collection of 626 receipts for
  training and 347 receipts for testing).
- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of
  400,000 images belonging to one of 16 classes).

The abstract from the paper is the following:

*Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the
widespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,
while neglecting layout and style information that is vital for document image understanding. In this paper, we propose
the LayoutLM to jointly model interactions between text and layout information across scanned document images, which is
beneficial for a great number of real-world document image understanding tasks such as information extraction from
scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM.
To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for
document-level pretraining. It achieves new state-of-the-art results in several downstream tasks, including form
understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification
(from 93.07 to 94.42).*

## Usage tips

- In addition to *input_ids*, [`~transformers.LayoutLMModel.forward`] also expects the input `bbox`, which are
  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such
  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where
  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the
  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000
  scale.


key: _node_content
string_value:
{"id_": "bd69daf8-86df-4163-97d0-01ecb98410a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef8b91f4-bc6a-4711-adc7-07b4c541a650", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd04cbee546fc1ffa9670497c6a1c019aca937c65cfdec8c905d64deb20f251", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4f3893d-8ead-4293-8588-67d0d7c3c235", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "2faf7b55caca22ddbd5fa27a00fa0f15324178c0e972ddfc3721b8f621b182c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9ef915e-dada-4dda-b865-365035d45b3e", "node_type": "1", "metadata": {}, "hash": "a3d8f9c18c0d0ecda730ba4098dd00f999f0471a8239e6b17aed453831835dfc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1156, "end_char_idx": 3859, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Despite the
widespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,
while neglecting layout and style information that is vital for document image understanding. In this paper, we propose
the LayoutLM to jointly model interactions between text and layout information across scanned document images, which is
beneficial for a great number of real-world document image understanding tasks such as information extraction from
scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM.
To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for
document-level pretraining. It achieves new state-of-the-art results in several downstream tasks, including form
understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification
(from 93.07 to 94.42).*

## Usage tips

- In addition to *input_ids*, [`~transformers.LayoutLMModel.forward`] also expects the input `bbox`, which are
  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such
  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where
  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the
  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000
  scale. To normalize, you can use the following function:

```python
def normalize_bbox(bbox, width, height):
    return [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
```

Here, `width` and `height` correspond to the width and height of the original document in which the token
occurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:

```python
from PIL import Image

# Document can be a png, jpg, etc. PDFs must be converted to images.
image = Image.open(name_of_your_document).convert("RGB")

width, height = image.size
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "a9ef915e-dada-4dda-b865-365035d45b3e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef8b91f4-bc6a-4711-adc7-07b4c541a650", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd04cbee546fc1ffa9670497c6a1c019aca937c65cfdec8c905d64deb20f251", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bd69daf8-86df-4163-97d0-01ecb98410a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "af7d8b562821a5f551a4bde51033a3740fb5e92578c9ca5780c502acc29c2b41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2321, "end_char_idx": 6448, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These can be obtained using an external OCR engine such
  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where
  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the
  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000
  scale. To normalize, you can use the following function:

```python
def normalize_bbox(bbox, width, height):
    return [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
```

Here, `width` and `height` correspond to the width and height of the original document in which the token
occurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:

```python
from PIL import Image

# Document can be a png, jpg, etc. PDFs must be converted to images.
image = Image.open(name_of_your_document).convert("RGB")

width, height = image.size
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


<PipelineTag pipeline="document-question-answering" />

- A blog post on [fine-tuning
  LayoutLM for document-understanding using Keras & Hugging Face
  Transformers](https://www.philschmid.de/fine-tuning-layoutlm-keras).

- A blog post on how to [fine-tune LayoutLM for document-understanding using only Hugging Face Transformers](https://www.philschmid.de/fine-tuning-layoutlm).

- A notebook on how to [fine-tune LayoutLM on the FUNSD dataset with image embeddings](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Add_image_embeddings_to_LayoutLM.ipynb).

- See also: [Document question answering task guide](../tasks/document_question_answering)

<PipelineTag pipeline="text-classification" />

- A notebook on how to [fine-tune LayoutLM for sequence classification on the RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification" />

- A notebook on how to [ fine-tune LayoutLM for token classification on the FUNSD dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb).
- [Token classification task guide](../tasks/token_classification)

**Other resources**
- [Masked language modeling task guide](../tasks/masked_language_modeling)

🚀 Deploy

- A blog post on how to [Deploy LayoutLM with Hugging Face Inference Endpoints](https://www.philschmid.de/inference-endpoints-layoutlm).

## LayoutLMConfig

[[autodoc]] LayoutLMConfig

## LayoutLMTokenizer

[[autodoc]] LayoutLMTokenizer

## LayoutLMTokenizerFast

[[autodoc]] LayoutLMTokenizerFast

<frameworkcontent>
<pt>

## LayoutLMModel

[[autodoc]] LayoutLMModel

## LayoutLMForMaskedLM

[[autodoc]] LayoutLMForMaskedLM

## LayoutLMForSequenceClassification

[[autodoc]] LayoutLMForSequenceClassification

## LayoutLMForTokenClassification

[[autodoc]] LayoutLMForTokenClassification

## LayoutLMForQuestionAnswering

[[autodoc]] LayoutLMForQuestionAnswering

</pt>
<tf>

## TFLayoutLMModel

[[autodoc]] TFLayoutLMModel

## TFLayoutLMForMaskedLM

[[autodoc]] TFLayoutLMForMaskedLM

## TFLayoutLMForSequenceClassification

[[autodoc]] TFLayoutLMForSequenceClassification

## TFLayoutLMForTokenClassification

[[autodoc]] TFLayoutLMForTokenClassification

## TFLayoutLMForQuestionAnswering

[[autodoc]] TFLayoutLMForQuestionAnswering

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "27ba1089-6890-4c2d-afb2-a4297cd5a875", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/table-transformer", "title": "Table Transformer", "tokens": 839, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4c92d06b-af2f-40f1-9e92-7570162e83de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/table-transformer", "title": "Table Transformer", "tokens": 839, "retrieve_doc": true, "source": "transformers"}, "hash": "fb46a3182597b5fb1e5e9da231e9f3dc48ddd06f9afa479cb8e52486178b0c57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Table Transformer

## Overview

The Table Transformer model was proposed in [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://arxiv.org/abs/2110.00061) by
Brandon Smock, Rohith Pesala, Robin Abraham. The authors introduce a new dataset, PubTables-1M, to benchmark progress in table extraction from unstructured documents,
as well as table structure recognition and functional analysis. The authors train 2 [DETR](detr) models, one for table detection and one for table structure recognition, dubbed Table Transformers.

The abstract from the paper is the following:

*Recently, significant progress has been made applying machine learning to the problem of table structure inference and extraction from unstructured documents.
However, one of the greatest challenges remains the creation of datasets with complete, unambiguous ground truth at scale. To address this, we develop a new, more
comprehensive dataset for table extraction, called PubTables-1M. PubTables-1M contains nearly one million tables from scientific articles, supports multiple input
modalities, and contains detailed header and location information for table structures, making it useful for a wide variety of modeling approaches. It also addresses a significant
source of ground truth inconsistency observed in prior datasets called oversegmentation, using a novel canonicalization procedure. We demonstrate that these improvements lead to a
significant increase in training performance and a more reliable estimate of model performance at evaluation for table structure recognition. Further, we show that transformer-based
object detection models trained on PubTables-1M produce excellent results for all three tasks of detection, structure recognition, and functional analysis without the need for any
special customization for these tasks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/table_transformer_architecture.jpeg"
alt="drawing" width="600"/>

<small> Table detection and table structure recognition clarified. Taken from the <a href="https://arxiv.org/abs/2110.00061">original paper</a>. </small>

The authors released 2 models, one for [table detection](https://huggingface.co/microsoft/table-transformer-detection) in 
documents, one for [table structure recognition](https://huggingface.co/microsoft/table-transformer-structure-recognition) 
(the task of recognizing the individual rows, columns etc. in a table).

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be
found [here](https://github.com/microsoft/table-transformer).

## Resources

<PipelineTag pipeline="object-detection"/>

- A demo notebook for the Table Transformer can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Table%20Transformer).
- It turns out padding of images is quite important for detection. An interesting Github thread with replies from the authors can be found [here](https://github.com/microsoft/table-transformer/issues/68).

## TableTransformerConfig

[[autodoc]] TableTransformerConfig

## TableTransformerModel

[[autodoc]] TableTransformerModel
    - forward

## TableTransformerForObjectDetection

[[autodoc]] TableTransformerForObjectDetection
    - forward


key: _node_content
string_value:
{"id_": "8da19a9e-cc27-446f-afc2-10e1bd84f02a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "00b16b0e-2fbe-4dfb-94b8-2e177699fcd8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "c3586208308d2f8147c299a64b332cb34ba08c8ef2856d537908a5d7975f1fe2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "339e1450-fb37-4151-9c06-a0ad0e2f93c8", "node_type": "1", "metadata": {}, "hash": "65a7dca4b06655e142721d57bda860399cb9caa0f2ec0ed005b35822edc88417", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3229, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LiLT

## Overview

The LiLT model was proposed in [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://arxiv.org/abs/2202.13669) by Jiapeng Wang, Lianwen Jin, Kai Ding.
LiLT allows to combine any pre-trained RoBERTa text encoder with a lightweight Layout Transformer, to enable [LayoutLM](layoutlm)-like document understanding for many
languages.

The abstract from the paper is the following:

*Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/lilt_architecture.jpg"
alt="drawing" width="600"/>

<small> LiLT architecture. Taken from the <a href="https://arxiv.org/abs/2202.13669">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/jpwang/lilt).

## Usage tips

- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).
The script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):

```python
from transformers import LiltModel

model = LiltModel.from_pretrained("path_to_your_files")
model.push_to_hub("name_of_repo_on_the_hub")
```

- When preparing data for the model, make sure to use the token vocabulary that corresponds to the RoBERTa checkpoint you combined with the Layout Transformer.
- As [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) uses the same vocabulary as [LayoutLMv3](layoutlmv3), one can use [`LayoutLMv3TokenizerFast`] to prepare data for the model.
The same is true for [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-infoxlm-base): one can use [`LayoutXLMTokenizerFast`] for that model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LiLT.

- Demo notebooks for LiLT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LiLT).


key: _node_content
string_value:
{"id_": "339e1450-fb37-4151-9c06-a0ad0e2f93c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "00b16b0e-2fbe-4dfb-94b8-2e177699fcd8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "c3586208308d2f8147c299a64b332cb34ba08c8ef2856d537908a5d7975f1fe2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8da19a9e-cc27-446f-afc2-10e1bd84f02a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "5085f6e89221b332dc7fa8712a747c36c56d939f72db23189305c2ebe2ff1c34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1783, "end_char_idx": 4032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/jpwang/lilt).

## Usage tips

- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).
The script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):

```python
from transformers import LiltModel

model = LiltModel.from_pretrained("path_to_your_files")
model.push_to_hub("name_of_repo_on_the_hub")
```

- When preparing data for the model, make sure to use the token vocabulary that corresponds to the RoBERTa checkpoint you combined with the Layout Transformer.
- As [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) uses the same vocabulary as [LayoutLMv3](layoutlmv3), one can use [`LayoutLMv3TokenizerFast`] to prepare data for the model.
The same is true for [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-infoxlm-base): one can use [`LayoutXLMTokenizerFast`] for that model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LiLT.

- Demo notebooks for LiLT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LiLT).

**Documentation resources**
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## LiltConfig

[[autodoc]] LiltConfig

## LiltModel

[[autodoc]] LiltModel
    - forward

## LiltForSequenceClassification

[[autodoc]] LiltForSequenceClassification
    - forward

## LiltForTokenClassification

[[autodoc]] LiltForTokenClassification
    - forward

## LiltForQuestionAnswering

[[autodoc]] LiltForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "aae878e4-1db0-46e8-891b-1ac26acbff7f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89d7f3f-b706-49be-934d-5d3b81a256f6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "19b5f20d-8248-47d0-9e09-2b8e157bb7a7", "node_type": "1", "metadata": {}, "hash": "f2ed00b16198c0e1c433a3b2aceee0fd3c152faa917a9aeb417e6de4d34427ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3162, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# M2M100

## Overview

The M2M100 model was proposed in [Beyond English-Centric Multilingual Machine Translation](https://arxiv.org/abs/2010.11125) by Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky,
Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy
Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin.

The abstract from the paper is the following:

*Existing work in translation demonstrated the potential of massively multilingual machine translation by training a
single model able to translate between any pair of languages. However, much of this work is English-Centric by training
only on data which was translated from or to English. While this is supported by large sources of training data, it
does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation
model that can translate directly between any pair of 100 languages. We build and open source a training dataset that
covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how
to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters
to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly
translating between non-English directions while performing competitively to the best single systems of WMT. We
open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.*

This model was contributed by [valhalla](https://huggingface.co/valhalla).


## Usage tips and examples

M2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks. As the model is
multilingual it expects the sequences in a certain format: A special language id token is used as prefix in both the
source and target text. The source text format is `[lang_code] X [eos]`, where `lang_code` is source language
id for source text and target language id for target text, with `X` being the source or target text.

The [`M2M100Tokenizer`] depends on `sentencepiece` so be sure to install it before running the
examples. To install `sentencepiece` run `pip install sentencepiece`.

**Supervised Training**

```python
from transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer

model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="en", tgt_lang="fr")

src_text = "Life is like a box of chocolates."
tgt_text = "La vie est comme une boîte de chocolat."

model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")

loss = model(**model_inputs).loss  # forward pass
```

**Generation**

M2M100 uses the `eos_token_id` as the `decoder_start_token_id` for generation with the target language id 
being forced as the first generated token. To force the target language id as the first generated token, pass the 
*forced_bos_token_id* parameter to the *generate* method.


key: _node_content
string_value:
{"id_": "19b5f20d-8248-47d0-9e09-2b8e157bb7a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89d7f3f-b706-49be-934d-5d3b81a256f6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aae878e4-1db0-46e8-891b-1ac26acbff7f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "5cf8ef92fd3e09e4dc7e9b2434498fe68d7298b8c884d328831a4267c77eb675", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43189866-a8ec-4e0a-964c-0b9c6e8fb5cf", "node_type": "1", "metadata": {}, "hash": "4d1d84674c6d4a8e0a3c44dd72b321d370a79a8de96ad0844cbdd1f9a2422100", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1690, "end_char_idx": 4294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips and examples

M2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks. As the model is
multilingual it expects the sequences in a certain format: A special language id token is used as prefix in both the
source and target text. The source text format is `[lang_code] X [eos]`, where `lang_code` is source language
id for source text and target language id for target text, with `X` being the source or target text.

The [`M2M100Tokenizer`] depends on `sentencepiece` so be sure to install it before running the
examples. To install `sentencepiece` run `pip install sentencepiece`.

**Supervised Training**

```python
from transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer

model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="en", tgt_lang="fr")

src_text = "Life is like a box of chocolates."
tgt_text = "La vie est comme une boîte de chocolat."

model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")

loss = model(**model_inputs).loss  # forward pass
```

**Generation**

M2M100 uses the `eos_token_id` as the `decoder_start_token_id` for generation with the target language id 
being forced as the first generated token. To force the target language id as the first generated token, pass the 
*forced_bos_token_id* parameter to the *generate* method. The following example shows how to translate between 
Hindi to French and Chinese to English using the *facebook/m2m100_418M* checkpoint.

```python
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
>>> chinese_text = "生活就像一盒巧克力。"

>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

>>> # translate Hindi to French
>>> tokenizer.src_lang = "hi"
>>> encoded_hi = tokenizer(hi_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id("fr"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"La vie est comme une boîte de chocolat."

>>> # translate Chinese to English
>>> tokenizer.src_lang = "zh"
>>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"Life is like a box of chocolate."


key: _node_content
string_value:
{"id_": "43189866-a8ec-4e0a-964c-0b9c6e8fb5cf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89d7f3f-b706-49be-934d-5d3b81a256f6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "19b5f20d-8248-47d0-9e09-2b8e157bb7a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "4b8ab3fbbbc6eae92c63260fa533889f11ff753210100364612538dbdfb99f33", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "659607ed-deae-4f19-aee8-590dabf758bc", "node_type": "1", "metadata": {}, "hash": "e06ce49ff041b581891a189a4ed13b464ed7b51df174ef64942a966d09a09b3b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3033, "end_char_idx": 5700, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To force the target language id as the first generated token, pass the 
*forced_bos_token_id* parameter to the *generate* method. The following example shows how to translate between 
Hindi to French and Chinese to English using the *facebook/m2m100_418M* checkpoint.

```python
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
>>> chinese_text = "生活就像一盒巧克力。"

>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

>>> # translate Hindi to French
>>> tokenizer.src_lang = "hi"
>>> encoded_hi = tokenizer(hi_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id("fr"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"La vie est comme une boîte de chocolat."

>>> # translate Chinese to English
>>> tokenizer.src_lang = "zh"
>>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"Life is like a box of chocolate."
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## M2M100Config

[[autodoc]] M2M100Config

## M2M100Tokenizer

[[autodoc]] M2M100Tokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## M2M100Model

[[autodoc]] M2M100Model
    - forward

## M2M100ForConditionalGeneration

[[autodoc]] M2M100ForConditionalGeneration
    - forward

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.


key: _node_content
string_value:
{"id_": "659607ed-deae-4f19-aee8-590dabf758bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a89d7f3f-b706-49be-934d-5d3b81a256f6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43189866-a8ec-4e0a-964c-0b9c6e8fb5cf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "e3b98cc6792a447e0ecc270b2ca2fc8ce905e0dec6a682b770fcadcb8c33233f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4295, "end_char_idx": 6748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## M2M100Config

[[autodoc]] M2M100Config

## M2M100Tokenizer

[[autodoc]] M2M100Tokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## M2M100Model

[[autodoc]] M2M100Model
    - forward

## M2M100ForConditionalGeneration

[[autodoc]] M2M100ForConditionalGeneration
    - forward

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.

```python
>>> import torch
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda").eval()
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

>>> # translate Hindi to French
>>> hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
>>> tokenizer.src_lang = "hi"
>>> encoded_hi = tokenizer(hi_text, return_tensors="pt").to("cuda")
>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id("fr"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"La vie est comme une boîte de chocolat."
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp">
</div>


key: _node_content
string_value:
{"id_": "5eb6a53e-52de-41b6-adb2-45c418967c3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/instructblipvideo", "title": "InstructBlipVideo", "tokens": 901, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e94ec04a-aba1-4a57-9547-05c34acc3093", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/instructblipvideo", "title": "InstructBlipVideo", "tokens": 901, "retrieve_doc": true, "source": "transformers"}, "hash": "4a68ff5cb055c594da95aa27fba9e7922dac45edac3ab6c5dcca72acab03fac4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# InstructBlipVideo

## Overview

## Overview

The InstructBLIPVideo is an extension of the models proposed in [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500) by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.
InstructBLIPVideo uses the same architecture as [InstructBLIP](instructblip) and works with the same checkpoints as [InstructBLIP](instructblip). The only difference is the ability to process videos.

The abstract from the paper is the following:

*General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/instructblip_architecture.jpg"
alt="drawing" width="600"/>

<small> InstructBLIPVideo architecture. Taken from the <a href="https://arxiv.org/abs/2305.06500">original paper.</a> </small>

This model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).
The original code can be found [here](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).

## Usage tips

- The model was trained by sampling 4 frames per video, so it's recommended to sample 4 frames

## InstructBlipVideoConfig

[[autodoc]] InstructBlipVideoConfig
    - from_vision_qformer_text_configs

## InstructBlipVideoVisionConfig

[[autodoc]] InstructBlipVideoVisionConfig

## InstructBlipVideoQFormerConfig

[[autodoc]] InstructBlipVideoQFormerConfig

## InstructBlipVideoProcessor

[[autodoc]] InstructBlipVideoProcessor

## InstructBlipVideoImageProcessor

[[autodoc]] InstructBlipVideoImageProcessor
    - preprocess

## InstructBlipVideoVisionModel

[[autodoc]] InstructBlipVideoVisionModel
    - forward

## InstructBlipVideoQFormerModel

[[autodoc]] InstructBlipVideoQFormerModel
    - forward

## InstructBlipVideoForConditionalGeneration

[[autodoc]] InstructBlipVideoForConditionalGeneration
    - forward
    - generate


key: _node_content
string_value:
{"id_": "8bc92ddc-0ead-49ae-8e99-478d9a384b34", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "84fdac49-64d2-4405-8432-b6c335ead97d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "c837497986765448e67ed048d580f630604c9e5339ffb7ab5fdfbc216044e896", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f61f04f-e2b4-4a9c-ab71-5bd21e80844c", "node_type": "1", "metadata": {}, "hash": "c6faede5bac43651ab761145da80d9b11ae36c3346ba3b2d62900780f13b5e2c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OWLv2

## Overview

OWLv2 was proposed in [Scaling Open-Vocabulary Object Detection](https://arxiv.org/abs/2306.09683) by Matthias Minderer, Alexey Gritsenko, Neil Houlsby. OWLv2 scales up [OWL-ViT](owlvit) using self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. This results in large gains over the previous state-of-the-art for zero-shot object detection.

The abstract from the paper is the following:

*Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlv2_overview.png"
alt="drawing" width="600"/>

<small> OWLv2 high-level overview. Taken from the <a href="https://arxiv.org/abs/2306.09683">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage example

OWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images.


