# embedding_metadata

key: _node_content
string_value:
{"id_": "52d9d5e3-8628-4ec8-a0d3-9c69543c5cd5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1fabf5e2-7347-40a2-b484-9c741f1526d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a06facd3acf8d87f27a84db2737bb0ea9532b4d8ef61118efa840484cb54362", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "21916a5e-14be-491b-aa25-b738a9d45ae7", "node_type": "1", "metadata": {}, "hash": "31224a1abbd7ba94ccb28d25bb81b1aeacfa787a9a29eafa2a9f43532e08eae4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5080, "end_char_idx": 8138, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### 1. Use dedicated models

[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.
For example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: 

```python
>>> from transformers import SeamlessM4TForSpeechToSpeech
>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained("facebook/hf-seamless-m4t-medium")
```

Or you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.

```python
>>> from transformers import SeamlessM4TForTextToText
>>> model = SeamlessM4TForTextToText.from_pretrained("facebook/hf-seamless-m4t-medium")
```

Feel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.

#### 2. Change the speaker identity

You have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!

#### 3. Change the generation strategy

You can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.

#### 4. Generate speech and text at the same time

Use `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !

## Model architecture


SeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as "unit tokens," from the translated text.

Each modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.

Here's how the generation process works:

- Input text or speech is processed through its specific encoder.
- A decoder creates text tokens in the desired language.
- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.
- These unit tokens are then passed through the final vocoder to produce the actual speech.


This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## SeamlessM4TModel

[[autodoc]] SeamlessM4TModel
    - generate


## SeamlessM4TForTextToSpeech

[[autodoc]] SeamlessM4TForTextToSpeech
    - generate


## SeamlessM4TForSpeechToSpeech

[[autodoc]] SeamlessM4TForSpeechToSpeech
    - generate


key: _node_content
string_value:
{"id_": "21916a5e-14be-491b-aa25-b738a9d45ae7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "52d9d5e3-8628-4ec8-a0d3-9c69543c5cd5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6efe8c5cd5c617582c0df358c204650a01f4860ec6edd89161ffe88e3c5039d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6748, "end_char_idx": 9167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
SeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as "unit tokens," from the translated text.

Each modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.

Here's how the generation process works:

- Input text or speech is processed through its specific encoder.
- A decoder creates text tokens in the desired language.
- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.
- These unit tokens are then passed through the final vocoder to produce the actual speech.


This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## SeamlessM4TModel

[[autodoc]] SeamlessM4TModel
    - generate


## SeamlessM4TForTextToSpeech

[[autodoc]] SeamlessM4TForTextToSpeech
    - generate


## SeamlessM4TForSpeechToSpeech

[[autodoc]] SeamlessM4TForSpeechToSpeech
    - generate


## SeamlessM4TForTextToText

[[autodoc]] transformers.SeamlessM4TForTextToText
    - forward
    - generate

## SeamlessM4TForSpeechToText

[[autodoc]] transformers.SeamlessM4TForSpeechToText
    - forward
    - generate

## SeamlessM4TConfig

[[autodoc]] SeamlessM4TConfig


## SeamlessM4TTokenizer

[[autodoc]] SeamlessM4TTokenizer
    - __call__
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary


## SeamlessM4TTokenizerFast

[[autodoc]] SeamlessM4TTokenizerFast
    - __call__

## SeamlessM4TFeatureExtractor

[[autodoc]] SeamlessM4TFeatureExtractor
    - __call__

## SeamlessM4TProcessor

[[autodoc]] SeamlessM4TProcessor
    - __call__

## SeamlessM4TCodeHifiGan

[[autodoc]] SeamlessM4TCodeHifiGan


## SeamlessM4THifiGan

[[autodoc]] SeamlessM4THifiGan

## SeamlessM4TTextToUnitModel

[[autodoc]] SeamlessM4TTextToUnitModel

## SeamlessM4TTextToUnitForConditionalGeneration

[[autodoc]] SeamlessM4TTextToUnitForConditionalGeneration


key: _node_content
string_value:
{"id_": "3c1b76bf-1540-4273-868a-8f37cc72386f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "adb2c51d-8478-4bc1-974c-e44d5b94e168", "node_type": "1", "metadata": {}, "hash": "e20dafc9d6623ee78a309cc42aabc93d29c085b753dc87c9290905ac1c24940e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ImageGPT

## Overview

The ImageGPT model was proposed in [Generative Pretraining from Pixels](https://openai.com/blog/image-gpt) by Mark
Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever. ImageGPT (iGPT) is a GPT-2-like
model trained to predict the next pixel value, allowing for both unconditional and conditional image generation.

The abstract from the paper is the following:

*Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models
can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels,
without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels,
we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and
low-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide
ResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pre-trained models. We are also
competitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0%
top-1 accuracy on a linear probe of our features.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/imagegpt_architecture.png"
alt="drawing" width="600"/>

<small> Summary of the approach. Taken from the [original paper](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf). </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr), based on [this issue](https://github.com/openai/image-gpt/issues/7). The original code can be found
[here](https://github.com/openai/image-gpt).

## Usage tips

- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation
  function is used (namely "quick gelu"), and the layer normalization layers don't mean center the inputs. ImageGPT
  also doesn't have tied input- and output embeddings.
- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence
  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special "start of sentence" (SOS)
  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare
  images for the model.
- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
  performant image features useful for downstream tasks, such as image classification.


key: _node_content
string_value:
{"id_": "adb2c51d-8478-4bc1-974c-e44d5b94e168", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c1b76bf-1540-4273-868a-8f37cc72386f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1c903dbded76142f66fa13735b7f8315af44ff61c89e65ef3baab87084f13440", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "529294bd-5da9-4952-81b9-e76a7dbc5f9b", "node_type": "1", "metadata": {}, "hash": "5fccfa84855df62b8f1903509f9f52776a67d488581354f7e837cf743acf608c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1717, "end_char_idx": 3883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found
[here](https://github.com/openai/image-gpt).

## Usage tips

- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation
  function is used (namely "quick gelu"), and the layer normalization layers don't mean center the inputs. ImageGPT
  also doesn't have tied input- and output embeddings.
- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence
  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special "start of sentence" (SOS)
  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare
  images for the model.
- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
  performant image features useful for downstream tasks, such as image classification. The authors showed that the
  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as
  a sklearn logistic regression model for example). This is also referred to as "linear probing". Features can be
  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and
  then average-pool the hidden states at whatever layer you like.
- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can
  use [`ImageGPTForImageClassification`].
- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also
  train an XL variant, which they didn't release.


key: _node_content
string_value:
{"id_": "529294bd-5da9-4952-81b9-e76a7dbc5f9b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "adb2c51d-8478-4bc1-974c-e44d5b94e168", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6cbbee26956adc8a0e8b34c0882607d4ed370b04c41c28f5e86a846bbb23f6b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cc4da21d-0b98-405d-9982-dfa6b5425bb8", "node_type": "1", "metadata": {}, "hash": "42c5f4faa6c1ce043a0454ff6de7536d5d55d600c806f46e56d7872e218dedbd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2300, "end_char_idx": 4630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special "start of sentence" (SOS)
  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare
  images for the model.
- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
  performant image features useful for downstream tasks, such as image classification. The authors showed that the
  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as
  a sklearn logistic regression model for example). This is also referred to as "linear probing". Features can be
  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and
  then average-pool the hidden states at whatever layer you like.
- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can
  use [`ImageGPTForImageClassification`].
- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also
  train an XL variant, which they didn't release. The differences in size are summarized in the following table:

| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |
|---|---|---|---|---|---|
| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |
| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |
| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |
| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |
| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |
| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ImageGPT.


key: _node_content
string_value:
{"id_": "cc4da21d-0b98-405d-9982-dfa6b5425bb8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "529294bd-5da9-4952-81b9-e76a7dbc5f9b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "db95983905acb49a976ff2b4ed598080f1973a87f23852e3790e8dac13d436a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3813, "end_char_idx": 5858, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The authors did also
  train an XL variant, which they didn't release. The differences in size are summarized in the following table:

| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |
|---|---|---|---|---|---|
| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |
| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |
| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |
| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |
| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |
| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ImageGPT.

<PipelineTag pipeline="image-classification"/>

- Demo notebooks for ImageGPT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ImageGPT).
- [`ImageGPTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ImageGPTConfig

[[autodoc]] ImageGPTConfig

## ImageGPTFeatureExtractor

[[autodoc]] ImageGPTFeatureExtractor
    - __call__

## ImageGPTImageProcessor

[[autodoc]] ImageGPTImageProcessor
    - preprocess

## ImageGPTModel

[[autodoc]] ImageGPTModel
    - forward

## ImageGPTForCausalImageModeling

[[autodoc]] ImageGPTForCausalImageModeling
    - forward

## ImageGPTForImageClassification

[[autodoc]] ImageGPTForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "d5ea83f8-899a-4cf7-ab9a-70480f467a63", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nezha/", "title": "Nezha", "tokens": 857, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7fca0363-491d-4a12-982b-f88fee9d1dc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nezha/", "title": "Nezha", "tokens": 857, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9133059a4c28828a08cbe8358c36f2e56f7596e1810d73b28d9b51f2d8a0a3bf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2821, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Nezha

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The Nezha model was proposed in [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://arxiv.org/abs/1909.00204) by Junqiu Wei et al.

The abstract from the paper is the following:

*The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks
due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.
In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed
representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks.
The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional
Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy,
Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA
achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including
named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti)
and natural language inference (XNLI).*

This model was contributed by [sijunhe](https://huggingface.co/sijunhe). The original code can be found [here](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-PyTorch).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## NezhaConfig

[[autodoc]] NezhaConfig

## NezhaModel

[[autodoc]] NezhaModel
    - forward

## NezhaForPreTraining

[[autodoc]] NezhaForPreTraining
    - forward

## NezhaForMaskedLM

[[autodoc]] NezhaForMaskedLM
    - forward

## NezhaForNextSentencePrediction

[[autodoc]] NezhaForNextSentencePrediction
    - forward

## NezhaForSequenceClassification

[[autodoc]] NezhaForSequenceClassification
    - forward

## NezhaForMultipleChoice

[[autodoc]] NezhaForMultipleChoice
    - forward

## NezhaForTokenClassification

[[autodoc]] NezhaForTokenClassification
    - forward

## NezhaForQuestionAnswering

[[autodoc]] NezhaForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "c1885d98-77db-40e1-9cf1-02f320e05b8a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c60a644fba1fa1a6d31884cc274859929434cc364eb0fa31f56b8f8ed0726a90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "40ff3b4e-6223-4539-bc91-aaf28c4d01a5", "node_type": "1", "metadata": {}, "hash": "d4e4b6db29d7225bf740abda8c406d28516077dde573d58a7868566c3b43b491", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Audio Spectrogram Transformer

## Overview

The Audio Spectrogram Transformer model was proposed in [AST: Audio Spectrogram Transformer](https://arxiv.org/abs/2104.01778) by Yuan Gong, Yu-An Chung, James Glass.
The Audio Spectrogram Transformer applies a [Vision Transformer](vit) to audio, by turning audio into an image (spectrogram). The model obtains state-of-the-art results
for audio classification.

The abstract from the paper is the following:

*In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png"
alt="drawing" width="600"/>

<small> Audio Spectrogram Transformer architecture. Taken from the <a href="https://arxiv.org/abs/2104.01778">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/YuanGongND/ast).

## Usage tips

- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make
sure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet
mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how
the authors compute the stats for a downstream dataset.
- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the
[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.


key: _node_content
string_value:
{"id_": "40ff3b4e-6223-4539-bc91-aaf28c4d01a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c60a644fba1fa1a6d31884cc274859929434cc364eb0fa31f56b8f8ed0726a90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c1885d98-77db-40e1-9cf1-02f320e05b8a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8a182a06b3f56a8fcdb9edef652a01ffa28a98f2681e7b284cbd46f3436eff9f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec6dcd84-36d9-4f9c-bc18-11eb71d6baf2", "node_type": "1", "metadata": {}, "hash": "d901af1d1cdc7253b73f04dc6ef84ddb1b30a0d78d7c3f3bc7d5c81caeba8116", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1651, "end_char_idx": 4891, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2104.01778">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/YuanGongND/ast).

## Usage tips

- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make
sure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet
mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how
the authors compute the stats for a downstream dataset.
- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the
[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ASTForAudioClassification
model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        27 |                                         6 |                      4.5 |
|            2 |                                        12 |                                         6 |                      2   |
|            4 |                                        21 |                                         8 |                      2.62 |
|            8 |                                        40 |                                        14 |                      2.86 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with the Audio Spectrogram Transformer.


key: _node_content
string_value:
{"id_": "ec6dcd84-36d9-4f9c-bc18-11eb71d6baf2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c60a644fba1fa1a6d31884cc274859929434cc364eb0fa31f56b8f8ed0726a90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "40ff3b4e-6223-4539-bc91-aaf28c4d01a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "00aaae614a4ee9f8b0a250f17e6f06f4b1ef64309acbebdae80f2f8869172477", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3233, "end_char_idx": 5926, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ASTForAudioClassification
model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        27 |                                         6 |                      4.5 |
|            2 |                                        12 |                                         6 |                      2   |
|            4 |                                        21 |                                         8 |                      2.62 |
|            8 |                                        40 |                                        14 |                      2.86 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with the Audio Spectrogram Transformer.

<PipelineTag pipeline="audio-classification"/>

- A notebook illustrating inference with AST for audio classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/AST).
- [`ASTForAudioClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).
- See also: [Audio classification](../tasks/audio_classification).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ASTConfig

[[autodoc]] ASTConfig

## ASTFeatureExtractor

[[autodoc]] ASTFeatureExtractor
    - __call__

## ASTModel

[[autodoc]] ASTModel
    - forward

## ASTForAudioClassification

[[autodoc]] ASTForAudioClassification
    - forward


key: _node_content
string_value:
{"id_": "55bdca4b-6e0a-479a-8897-c0ac019ad574", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1e405b7a09187cfb7a031c46e362465ca0df8ae5f29ad7dfe8f42b1250880ab2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1b9c3d13-4dd6-44d1-9fab-a5b33f605c30", "node_type": "1", "metadata": {}, "hash": "5b4286353af35aec9c5e32c51e60427e675483b5392132ad419be9ff6b529f1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Mask2Former

## Overview

The Mask2Former model was proposed in [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527) by Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar. Mask2Former is a unified framework for panoptic, instance and semantic segmentation and features significant performance and efficiency improvements over [MaskFormer](maskformer).

The abstract from the paper is the following:

*Image segmentation groups pixels with different semantics, e.g., category or instance membership. Each choice
of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mask2former_architecture.jpg" alt="drawing" width="600"/>

<small> Mask2Former architecture. Taken from the <a href="https://arxiv.org/abs/2112.01527">original paper.</a> </small>

This model was contributed by [Shivalika Singh](https://huggingface.co/shivi) and [Alara Dirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/facebookresearch/Mask2Former).

## Usage tips

- Mask2Former uses the same preprocessing and postprocessing steps as [MaskFormer](maskformer). Use [`Mask2FormerImageProcessor`] or [`AutoImageProcessor`] to prepare images and optional targets for the model.
- To get the final segmentation, depending on the task, you can call [`~Mask2FormerImageProcessor.post_process_semantic_segmentation`] or [`~Mask2FormerImageProcessor.post_process_instance_segmentation`] or [`~Mask2FormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`Mask2FormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Mask2Former.

- Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Mask2Former).
- Scripts for finetuning [`Mask2Former`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "1b9c3d13-4dd6-44d1-9fab-a5b33f605c30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1e405b7a09187cfb7a031c46e362465ca0df8ae5f29ad7dfe8f42b1250880ab2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55bdca4b-6e0a-479a-8897-c0ac019ad574", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "18e4557c3f03019fecefae6c35de9847bfc365a71e7ef6640d0c75713d66f524", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1832, "end_char_idx": 4062, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/facebookresearch/Mask2Former).

## Usage tips

- Mask2Former uses the same preprocessing and postprocessing steps as [MaskFormer](maskformer). Use [`Mask2FormerImageProcessor`] or [`AutoImageProcessor`] to prepare images and optional targets for the model.
- To get the final segmentation, depending on the task, you can call [`~Mask2FormerImageProcessor.post_process_semantic_segmentation`] or [`~Mask2FormerImageProcessor.post_process_instance_segmentation`] or [`~Mask2FormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`Mask2FormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Mask2Former.

- Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Mask2Former).
- Scripts for finetuning [`Mask2Former`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.

## Mask2FormerConfig

[[autodoc]] Mask2FormerConfig

## MaskFormer specific outputs

[[autodoc]] models.mask2former.modeling_mask2former.Mask2FormerModelOutput

[[autodoc]] models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput

## Mask2FormerModel

[[autodoc]] Mask2FormerModel
    - forward

## Mask2FormerForUniversalSegmentation

[[autodoc]] Mask2FormerForUniversalSegmentation
    - forward

## Mask2FormerImageProcessor

[[autodoc]] Mask2FormerImageProcessor
    - preprocess
    - encode_inputs
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation


key: _node_content
string_value:
{"id_": "856247ce-1467-4a96-a1b7-8804e881bb32", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2f553e422e3f914165488599c6bf47f8ff836bf1d40842ef08ca56b9b6c3d94c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f05d248b-ad1a-4e66-985e-834db5cfc6db", "node_type": "1", "metadata": {}, "hash": "d8fa86f6e1ff552031aaa60b886f2bb06b82d080b1961a69ca32028a0f30db33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3117, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PatchTSMixer

## Overview

The PatchTSMixer model was proposed in [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://arxiv.org/pdf/2306.09364.pdf) by Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong and Jayant Kalagnanam.


PatchTSMixer is a lightweight time-series modeling approach based on the MLP-Mixer architecture. In this HuggingFace implementation, we provide PatchTSMixer's capabilities to effortlessly facilitate lightweight mixing across patches, channels, and hidden features for effective multivariate time-series modeling. It also supports various attention mechanisms starting from simple gated attention to more complex self-attention blocks that can be customized accordingly. The model can be pretrained and subsequently used for various downstream tasks such as forecasting, classification and regression.


The abstract from the paper is the following:

*TSMixer is a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules designed for multivariate forecasting and representation learning on patched time series. Our model draws inspiration from the success of MLP-Mixer models in computer vision. We demonstrate the challenges involved in adapting Vision MLP-Mixer for time series and introduce empirically validated components to enhance accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a Hybrid channel modeling approach to effectively handle noisy channel interactions and generalization across diverse datasets, a common challenge in existing patch channel-mixing methods. Additionally, a simple gated attention mechanism is introduced in the backbone to prioritize important features. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X).*

This model was contributed by [ajati](https://huggingface.co/ajati), [vijaye12](https://huggingface.co/vijaye12), 
[gsinthong](https://huggingface.co/gsinthong), [namctin](https://huggingface.co/namctin),
[wmgifford](https://huggingface.co/wmgifford), [kashif](https://huggingface.co/kashif).

## Usage example

The code snippet below shows how to randomly initialize a PatchTSMixer model. The model is compatible with the [Trainer API](../trainer.md).

```python

from transformers import PatchTSMixerConfig, PatchTSMixerForPrediction
from transformers import Trainer, TrainingArguments,


key: _node_content
string_value:
{"id_": "f05d248b-ad1a-4e66-985e-834db5cfc6db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2f553e422e3f914165488599c6bf47f8ff836bf1d40842ef08ca56b9b6c3d94c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "856247ce-1467-4a96-a1b7-8804e881bb32", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "42a730664e9e703e874fed0bc3bac443ad211f2ab402eb3591b9e5bec0c195ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3120, "end_char_idx": 4296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
config = PatchTSMixerConfig(context_length = 512, prediction_length = 96)
model = PatchTSMixerForPrediction(config)
trainer = Trainer(model=model, args=training_args, 
            train_dataset=train_dataset,
            eval_dataset=valid_dataset)
trainer.train()
results = trainer.evaluate(test_dataset)
```

## Usage tips

The model can also be used for time series classification and time series regression. See the respective [`PatchTSMixerForTimeSeriesClassification`] and [`PatchTSMixerForRegression`] classes.

## Resources

- A blog post explaining PatchTSMixer in depth can be found [here](https://huggingface.co/blog/patchtsmixer). The blog can also be opened in Google Colab.

## PatchTSMixerConfig

[[autodoc]] PatchTSMixerConfig


## PatchTSMixerModel

[[autodoc]] PatchTSMixerModel
    - forward


## PatchTSMixerForPrediction

[[autodoc]] PatchTSMixerForPrediction
    - forward


## PatchTSMixerForTimeSeriesClassification

[[autodoc]] PatchTSMixerForTimeSeriesClassification
    - forward


## PatchTSMixerForPretraining

[[autodoc]] PatchTSMixerForPretraining
    - forward


## PatchTSMixerForRegression

[[autodoc]] PatchTSMixerForRegression
    - forward


key: _node_content
string_value:
{"id_": "0abe423b-c2eb-4095-972d-80ddacbdb8e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "640543e6-9609-4f07-bead-3efadc87471a", "node_type": "1", "metadata": {}, "hash": "1dd7787d4c5d5d1358e50d40eeb33a44e315385e0003840c427830e461f7a0f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3117, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPTBigCode

## Overview

The GPTBigCode model was proposed in [SantaCoder: don't reach for the stars!](https://arxiv.org/abs/2301.03988) by BigCode. The listed authors are: Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra.

The abstract from the paper is the following:

*The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at [this https URL.](https://huggingface.co/bigcode)*

The model is an optimized [GPT2 model](https://huggingface.co/docs/transformers/model_doc/gpt2) with support for Multi-Query Attention.

## Implementation details

The main differences compared to GPT2.
- Added support for Multi-Query Attention.
- Use `gelu_pytorch_tanh` instead of classic `gelu`.
- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).
- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).
- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`
- Cache the attention mask value to avoid recreating it every time.
- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.
- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.
- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)


key: _node_content
string_value:
{"id_": "640543e6-9609-4f07-bead-3efadc87471a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0abe423b-c2eb-4095-972d-80ddacbdb8e5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "60443551a8bec698603c33b7303e3cb390e9efd90e16cf52146cf04fdf989ebd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f7d5090-7ee9-4574-8ee7-703e43448ca8", "node_type": "1", "metadata": {}, "hash": "0d3cbefe0687d8c467dd8b5332bbc78467db7c65dda39530751c69ff35c60ab5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1483, "end_char_idx": 3965, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at [this https URL.](https://huggingface.co/bigcode)*

The model is an optimized [GPT2 model](https://huggingface.co/docs/transformers/model_doc/gpt2) with support for Multi-Query Attention.

## Implementation details

The main differences compared to GPT2.
- Added support for Multi-Query Attention.
- Use `gelu_pytorch_tanh` instead of classic `gelu`.
- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).
- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).
- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`
- Cache the attention mask value to avoid recreating it every time.
- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.
- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.
- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)
- Use the memory layout (self.num_heads, 3, self.head_dim) instead of `(3, self.num_heads, self.head_dim)` for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible with the original openai-community/gpt2 model).

You can read more about the optimizations in the [original pull request](https://github.com/huggingface/transformers/pull/22575)

## Combining Starcoder and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g.


key: _node_content
string_value:
{"id_": "5f7d5090-7ee9-4574-8ee7-703e43448ca8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "640543e6-9609-4f07-bead-3efadc87471a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7da721b38550c1bd7717d317f047ea13002067d150452734ef04e16e4fe47c37", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e5ea052-956b-43d1-a50f-36b35ede398b", "node_type": "1", "metadata": {}, "hash": "bc0bfe69c72e629cb05830c35b3b9cb24f0283d0bdaaa27237d12bbde2a8b099", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2271, "end_char_idx": 5241, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Added support for Multi-Query Attention.
- Use `gelu_pytorch_tanh` instead of classic `gelu`.
- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).
- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).
- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`
- Cache the attention mask value to avoid recreating it every time.
- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.
- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.
- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)
- Use the memory layout (self.num_heads, 3, self.head_dim) instead of `(3, self.num_heads, self.head_dim)` for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible with the original openai-community/gpt2 model).

You can read more about the optimizations in the [original pull request](https://github.com/huggingface/transformers/pull/22575)

## Combining Starcoder and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("bigcode/gpt_bigcode-santacoder", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("bigcode/gpt_bigcode-santacoder")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)
>>> tokenizer.batch_decode(generated_ids)[0]
'def hello_world():\n    print("hello world")\n\nif __name__ == "__main__":\n    print("hello world")\n<|endoftext|>'
```

### Expected speedups

Below is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `bigcode/starcoder` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/starcoder-speedup.png">
</div>


key: _node_content
string_value:
{"id_": "7e5ea052-956b-43d1-a50f-36b35ede398b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f7d5090-7ee9-4574-8ee7-703e43448ca8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e1fc7e37ce1e93eeed7e56705f61dda3588ad7a4b0a9545c19b35da4419a782b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3690, "end_char_idx": 5631, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("bigcode/gpt_bigcode-santacoder", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("bigcode/gpt_bigcode-santacoder")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)
>>> tokenizer.batch_decode(generated_ids)[0]
'def hello_world():\n    print("hello world")\n\nif __name__ == "__main__":\n    print("hello world")\n<|endoftext|>'
```

### Expected speedups

Below is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `bigcode/starcoder` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/starcoder-speedup.png">
</div>


## GPTBigCodeConfig

[[autodoc]] GPTBigCodeConfig

## GPTBigCodeModel

[[autodoc]] GPTBigCodeModel
    - forward

## GPTBigCodeForCausalLM

[[autodoc]] GPTBigCodeForCausalLM
    - forward

## GPTBigCodeForSequenceClassification

[[autodoc]] GPTBigCodeForSequenceClassification
    - forward

## GPTBigCodeForTokenClassification

[[autodoc]] GPTBigCodeForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "150d2a9b-e139-4622-b166-17fb97910339", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nystromformer/", "title": "Nystr\u00f6mformer", "tokens": 796, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98051f69-9a2a-4bd6-9c66-4a2042b35beb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nystromformer/", "title": "Nystr\u00f6mformer", "tokens": 796, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "79839e0f1f69b61c039f942441047f9c7815d2f119980341a469d0458d1fb3d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2705, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Nyströmformer

## Overview

The Nyströmformer model was proposed in [*Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention*](https://arxiv.org/abs/2102.03902) by Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn
Fung, Yin Li, and Vikas Singh.

The abstract from the paper is the following:

*Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component
that drives the impressive performance of Transformers is the self-attention mechanism that encodes the influence or
dependence of other tokens on each specific token. While beneficial, the quadratic complexity of self-attention on the
input sequence length has limited its application to longer sequences -- a topic being actively studied in the
community. To address this limitation, we propose Nyströmformer -- a model that exhibits favorable scalability as a
function of sequence length. Our idea is based on adapting the Nyström method to approximate standard self-attention
with O(n) complexity. The scalability of Nyströmformer enables application to longer sequences with thousands of
tokens. We perform evaluations on multiple downstream tasks on the GLUE benchmark and IMDB reviews with standard
sequence length, and find that our Nyströmformer performs comparably, or in a few cases, even slightly better, than
standard self-attention. On longer sequence tasks in the Long Range Arena (LRA) benchmark, Nyströmformer performs
favorably relative to other efficient self-attention methods. Our code is available at this https URL.*

This model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/Nystromformer).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## NystromformerConfig

[[autodoc]] NystromformerConfig

## NystromformerModel

[[autodoc]] NystromformerModel
    - forward

## NystromformerForMaskedLM

[[autodoc]] NystromformerForMaskedLM
    - forward

## NystromformerForSequenceClassification

[[autodoc]] NystromformerForSequenceClassification
    - forward

## NystromformerForMultipleChoice

[[autodoc]] NystromformerForMultipleChoice
    - forward

## NystromformerForTokenClassification

[[autodoc]] NystromformerForTokenClassification
    - forward

## NystromformerForQuestionAnswering

[[autodoc]] NystromformerForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "51c98a32-bc2d-4536-b21a-dc0bd19e5b13", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/glpn/", "title": "GLPN", "tokens": 895, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9cd5b097-c0ee-4633-9938-bb8949124f82", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/glpn/", "title": "GLPN", "tokens": 895, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "414b5f3674d24a0aabce2e5beabfa71ab84e94e3423c05fecd044cda0b06ac3d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GLPN

<Tip>

This is a recently introduced model so the API hasn't been tested extensively. There may be some bugs or slight
breaking changes to fix it in the future. If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title).

</Tip>

## Overview

The GLPN model was proposed in [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436)  by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim.
GLPN combines [SegFormer](segformer)'s hierarchical mix-Transformer with a lightweight decoder for monocular depth estimation. The proposed decoder shows better performance than the previously proposed decoders, with considerably
less computational complexity.

The abstract from the paper is the following:

*Depth estimation from a single image is an important task that can be applied to various fields in computer vision, and has grown rapidly with the development of convolutional neural networks. In this paper, we propose a novel structure and training strategy for monocular depth estimation to further improve the prediction accuracy of the network. We deploy a hierarchical transformer encoder to capture and convey the global context, and design a lightweight yet powerful decoder to generate an estimated depth map while considering local connectivity. By constructing connected paths between multi-scale local features and the global decoding stream with our proposed selective feature fusion module, the network can integrate both representations and recover fine details. In addition, the proposed decoder shows better performance than the previously proposed decoders, with considerably less computational complexity. Furthermore, we improve the depth-specific augmentation method by utilizing an important observation in depth estimation to enhance the model. Our network achieves state-of-the-art performance over the challenging depth dataset NYU Depth V2. Extensive experiments have been conducted to validate and show the effectiveness of the proposed approach. Finally, our model shows better generalisation ability and robustness than other comparative models.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg"
alt="drawing" width="600"/>

<small> Summary of the approach. Taken from the <a href="https://arxiv.org/abs/2201.07436" target="_blank">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/vinvino02/GLPDepth).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GLPN.

- Demo notebooks for [`GLPNForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GLPN).
- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)

## GLPNConfig

[[autodoc]] GLPNConfig

## GLPNFeatureExtractor

[[autodoc]] GLPNFeatureExtractor
    - __call__

## GLPNImageProcessor

[[autodoc]] GLPNImageProcessor
    - preprocess

## GLPNModel

[[autodoc]] GLPNModel
    - forward

## GLPNForDepthEstimation

[[autodoc]] GLPNForDepthEstimation
    - forward


key: _node_content
string_value:
{"id_": "1a395ab8-4203-4a51-9777-4200ee2f7c6e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gemma/", "title": "Gemma", "tokens": 661, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a2f21922-1938-4f7a-9c7e-460ddb249444", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gemma/", "title": "Gemma", "tokens": 661, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "acc214dff7a9ff1d4d6398eba07c40fd18a349bc40bb24f6128a87dc81ae78db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1970, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Gemma

## Overview

The Gemma model was proposed in [Gemma: Open Models Based on Gemini Technology and Research](https://blog.google/technology/developers/gemma-open-models/) by Gemma Team, Google.
Gemma models are trained on 6T tokens, and released with 2 versions, 2b and 7b.

The abstract from the paper is the following:

*This work introduces Gemma, a new family of open language models demonstrating strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of our model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations*

Tips:

- The original checkpoints can be converted using the conversion script `src/transformers/models/gemma/convert_gemma_weights_to_hf.py` 

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Younes Belkada](https://huggingface.co/ybelkada), [Sanchit Gandhi](https://huggingface.co/sanchit-gandhi), [Pedro Cuenca](https://huggingface.co/pcuenq).


## GemmaConfig

[[autodoc]] GemmaConfig

## GemmaTokenizer

[[autodoc]] GemmaTokenizer


## GemmaTokenizerFast

[[autodoc]] GemmaTokenizerFast

## GemmaModel

[[autodoc]] GemmaModel
    - forward

## GemmaForCausalLM

[[autodoc]] GemmaForCausalLM
    - forward

## GemmaForSequenceClassification

[[autodoc]] GemmaForSequenceClassification
    - forward

## GemmaForTokenClassification

[[autodoc]] GemmaForTokenClassification
    - forward

## FlaxGemmaModel

[[autodoc]] FlaxGemmaModel
    - __call__

## FlaxGemmaForCausalLM

[[autodoc]] FlaxGemmaForCausalLM
    - __call__


key: _node_content
string_value:
{"id_": "08a9b906-fa64-4144-a9fe-9ffa9879a021", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wavlm/", "title": "WavLM", "tokens": 859, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4375e553-9e0d-4824-8b64-31a21aec6610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wavlm/", "title": "WavLM", "tokens": 859, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "76a15cd30edcb100bdc511288d2eea3c75c058998f9065c2d4b4603384847b22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2851, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# WavLM

## Overview

The WavLM model was proposed in [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://arxiv.org/abs/2110.13900) by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen,
Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu,
Michael Zeng, Furu Wei.

The abstract from the paper is the following:

*Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been
attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker
identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is
challenging. In this paper, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks.
WavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity
preservation. We first equip the Transformer structure with gated relative position bias to improve its capability on
recognition tasks. For better speaker discrimination, we propose an utterance mixing training strategy, where
additional overlapped utterances are created unsupervisedly and incorporated during model training. Lastly, we scale up
the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB
benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.*

Relevant checkpoints can be found under https://huggingface.co/models?other=wavlm.

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be
found [here](https://github.com/microsoft/unilm/tree/master/wavlm).

## Usage tips

- WavLM is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. Please use
  [`Wav2Vec2Processor`] for the feature extraction.
- WavLM model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].
- WavLM performs especially well on speaker verification, speaker identification, and speaker diarization tasks.

## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## WavLMConfig

[[autodoc]] WavLMConfig

## WavLMModel

[[autodoc]] WavLMModel
    - forward

## WavLMForCTC

[[autodoc]] WavLMForCTC
    - forward

## WavLMForSequenceClassification

[[autodoc]] WavLMForSequenceClassification
    - forward

## WavLMForAudioFrameClassification

[[autodoc]] WavLMForAudioFrameClassification
    - forward

## WavLMForXVector

[[autodoc]] WavLMForXVector
    - forward


key: _node_content
string_value:
{"id_": "f27ffbf5-b218-403a-a997-07f73a950c42", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/unispeech-sat/", "title": "UniSpeech-SAT", "tokens": 891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b141613c-b743-4fa7-88d1-f3f3e7bb41d8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/unispeech-sat/", "title": "UniSpeech-SAT", "tokens": 891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "44bd08737e3b53a1307a999b0ebbf5a66ec3d6e576354ebd27634ecdfc9f1d34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3262, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# UniSpeech-SAT

## Overview

The UniSpeech-SAT model was proposed in [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware
Pre-Training](https://arxiv.org/abs/2110.05752) by Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen,
Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu .

The abstract from the paper is the following:

*Self-supervised learning (SSL) is a long-standing goal for speech processing, since it utilizes large-scale unlabeled
data and avoids extensive human labeling. Recent years witness great successes in applying self-supervised learning in
speech recognition, while limited exploration was attempted in applying SSL for modeling speaker characteristics. In
this paper, we aim to improve the existing SSL framework for speaker representation learning. Two methods are
introduced for enhancing the unsupervised speaker information extraction. First, we apply the multi-task learning to
the current SSL framework, where we integrate the utterance-wise contrastive loss with the SSL objective function.
Second, for better speaker discrimination, we propose an utterance mixing strategy for data augmentation, where
additional overlapped utterances are created unsupervisedly and incorporate during training. We integrate the proposed
methods into the HuBERT framework. Experiment results on SUPERB benchmark show that the proposed system achieves
state-of-the-art performance in universal representation learning, especially for speaker identification oriented
tasks. An ablation study is performed verifying the efficacy of each proposed method. Finally, we scale up training
dataset to 94 thousand hours public audio data and achieve further performance improvement in all SUPERB tasks.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be
found [here](https://github.com/microsoft/UniSpeech/tree/main/UniSpeech-SAT).

## Usage tips

- UniSpeechSat is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
  Please use [`Wav2Vec2Processor`] for the feature extraction.
- UniSpeechSat model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be
  decoded using [`Wav2Vec2CTCTokenizer`].
- UniSpeechSat performs especially well on speaker verification, speaker identification, and speaker diarization tasks.

## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## UniSpeechSatConfig

[[autodoc]] UniSpeechSatConfig

## UniSpeechSat specific outputs

[[autodoc]] models.unispeech_sat.modeling_unispeech_sat.UniSpeechSatForPreTrainingOutput

## UniSpeechSatModel

[[autodoc]] UniSpeechSatModel
    - forward

## UniSpeechSatForCTC

[[autodoc]] UniSpeechSatForCTC
    - forward

## UniSpeechSatForSequenceClassification

[[autodoc]] UniSpeechSatForSequenceClassification
    - forward

## UniSpeechSatForAudioFrameClassification

[[autodoc]] UniSpeechSatForAudioFrameClassification
    - forward

## UniSpeechSatForXVector

[[autodoc]] UniSpeechSatForXVector
    - forward

## UniSpeechSatForPreTraining

[[autodoc]] UniSpeechSatForPreTraining
    - forward


key: _node_content
string_value:
{"id_": "90d68094-7547-44be-91ee-15c9a0a7dbfb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "56b9377c1e5aea4a1be7e9db377bc5b4cad9af43c2c12af715b989a47c124168", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e230e44-5da9-400c-a997-33ec7f426e22", "node_type": "1", "metadata": {}, "hash": "21851b0c561b8518650206f11e00571ae7bd1cd5fd205a0df870c75ce5f692e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DPR

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=dpr">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-dpr-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/dpr-question_encoder-bert-base-multilingual">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

Dense Passage Retrieval (DPR) is a set of tools and models for state-of-the-art open-domain Q&A research. It was
introduced in [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) by
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.

The abstract from the paper is the following:

*Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional
sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can
be practically implemented using dense representations alone, where embeddings are learned from a small number of
questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,
our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage
retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA
benchmarks.*

This model was contributed by [lhoestq](https://huggingface.co/lhoestq). The original code can be found [here](https://github.com/facebookresearch/DPR).

## Usage tips

- DPR consists in three models:

    * Question encoder: encode questions as vectors
    * Context encoder: encode contexts as vectors
    * Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question).


key: _node_content
string_value:
{"id_": "3e230e44-5da9-400c-a997-33ec7f426e22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "56b9377c1e5aea4a1be7e9db377bc5b4cad9af43c2c12af715b989a47c124168", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90d68094-7547-44be-91ee-15c9a0a7dbfb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9af9dcdb1ead3dda4a4c70f5268acfa4ebdea600f05c9d2603d2e94ee738ffd8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 512, "end_char_idx": 3067, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It was
introduced in [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) by
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.

The abstract from the paper is the following:

*Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional
sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can
be practically implemented using dense representations alone, where embeddings are learned from a small number of
questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,
our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage
retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA
benchmarks.*

This model was contributed by [lhoestq](https://huggingface.co/lhoestq). The original code can be found [here](https://github.com/facebookresearch/DPR).

## Usage tips

- DPR consists in three models:

    * Question encoder: encode questions as vectors
    * Context encoder: encode contexts as vectors
    * Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question).

## DPRConfig

[[autodoc]] DPRConfig

## DPRContextEncoderTokenizer

[[autodoc]] DPRContextEncoderTokenizer

## DPRContextEncoderTokenizerFast

[[autodoc]] DPRContextEncoderTokenizerFast

## DPRQuestionEncoderTokenizer

[[autodoc]] DPRQuestionEncoderTokenizer

## DPRQuestionEncoderTokenizerFast

[[autodoc]] DPRQuestionEncoderTokenizerFast

## DPRReaderTokenizer

[[autodoc]] DPRReaderTokenizer

## DPRReaderTokenizerFast

[[autodoc]] DPRReaderTokenizerFast

## DPR specific outputs

[[autodoc]] models.dpr.modeling_dpr.DPRContextEncoderOutput

[[autodoc]] models.dpr.modeling_dpr.DPRQuestionEncoderOutput

[[autodoc]] models.dpr.modeling_dpr.DPRReaderOutput

<frameworkcontent>
<pt>

## DPRContextEncoder

[[autodoc]] DPRContextEncoder
    - forward

## DPRQuestionEncoder

[[autodoc]] DPRQuestionEncoder
    - forward

## DPRReader

[[autodoc]] DPRReader
    - forward

</pt>
<tf>

## TFDPRContextEncoder

[[autodoc]] TFDPRContextEncoder
    - call

## TFDPRQuestionEncoder

[[autodoc]] TFDPRQuestionEncoder
    - call

## TFDPRReader

[[autodoc]] TFDPRReader
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "da9d2e2c-b742-4e68-8d56-6f802bbf8536", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/realm/", "title": "REALM", "tokens": 856, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "04ce72f2-eab1-44d9-bc35-0661dbb866f4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/realm/", "title": "REALM", "tokens": 856, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8cd9deb3f768c6ce0228ef2d2d000faac5d5b99296374d67c7aa03a71b8a4c30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# REALM

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The REALM model was proposed in [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909) by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang. It's a
retrieval-augmented language model that firstly retrieves documents from a textual knowledge corpus and then
utilizes retrieved documents to process question answering tasks.

The abstract from the paper is the following:

*Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks
such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network,
requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we
augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend
over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the
first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language
modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We
demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the
challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both
explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous
methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as
interpretability and modularity.*

This model was contributed by [qqaatw](https://huggingface.co/qqaatw). The original code can be found
[here](https://github.com/google-research/language/tree/master/language/realm).

## RealmConfig

[[autodoc]] RealmConfig

## RealmTokenizer

[[autodoc]] RealmTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary
    - batch_encode_candidates

## RealmTokenizerFast

[[autodoc]] RealmTokenizerFast
    - batch_encode_candidates

## RealmRetriever

[[autodoc]] RealmRetriever

## RealmEmbedder

[[autodoc]] RealmEmbedder
    - forward

## RealmScorer

[[autodoc]] RealmScorer
    - forward

## RealmKnowledgeAugEncoder

[[autodoc]] RealmKnowledgeAugEncoder
    - forward

## RealmReader

[[autodoc]] RealmReader
    - forward

## RealmForOpenQA

[[autodoc]] RealmForOpenQA
    - block_embedding_to
    - forward


key: _node_content
string_value:
{"id_": "d589a30b-957f-4f41-ad94-38f72e920d47", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "98b923fb-63d8-4301-bab1-898028454305", "node_type": "1", "metadata": {}, "hash": "ea5afae6ed9e4f976602b8ef2c2b78cf83f4a5d765f2c27de0b3f692d07bcc6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Video-LLaVA

## Overview

Video-LLaVa is an open-source multimodal LLM trained by fine-tuning LlamA/Vicuna on multimodal instruction-following data generated by Llava1.5 and VideChat. It is an auto-regressive language model, based on the transformer architecture. Video-LLaVa unifies visual representations to the language feature space, and enables an LLM to perform visual reasoning capabilities on both images and videos simultaneously.


The Video-LLaVA model was proposed in [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://arxiv.org/abs/2311.10122) by Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munang Ning, Peng Jin, Li Yuan.

The abstract from the paper is the following:

*The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in
visual-language understanding. Most existing approaches
encode images and videos into separate feature spaces,
which are then fed as inputs to large language models.
However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it
becomes challenging for a Large Language Model (LLM)
to learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA,
which learns from a mixed dataset of images and videos,
mutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4
image benchmark toolkits. Additionally, our Video-LLaVA
also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%,
and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that
Video-LLaVA mutually benefits images and videos within
a unified visual representation, outperforming models designed specifically for images or videos. We aim for this
work to provide modest insights into the multi-modal inputs
for the LLM*

## Usage tips:

- We advise users to use padding_side="left" when computing batched generation as it leads to more accurate results. Simply make sure to call processor.tokenizer.padding_side = "left" before generating.

- Note the model has not been explicitly trained to process multiple images/videos in the same prompt, although this is technically possible, you may experience inaccurate results.

- Note that the video inputs should have exactly 8 frames at the input, since the models were trained in that setting. 

This model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).
The original code can be found [here](https://github.com/PKU-YuanGroup/Video-LLaVA).


## Usage example

### Single Media Mode

The model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):

```python
import av
import torch
import numpy as np
from transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor

def read_video_pyav(container, indices):
    '''
    Decode the video with PyAV decoder.
    Args:
        container (`av.container.input.InputContainer`): PyAV container.
        indices (`List[int]`): List of frame indices to decode.


key: _node_content
string_value:
{"id_": "98b923fb-63d8-4301-bab1-898028454305", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d589a30b-957f-4f41-ad94-38f72e920d47", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "004f358524466c61ef5d6ac4d7a4bfdd4b20f6d6e9cbe32ea8cdc3104e3550a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4245e9a0-d00b-48f3-a00d-c12e253bbe6e", "node_type": "1", "metadata": {}, "hash": "e9855767cc8352d9cd036b2b8755bf1d71179ea38a63cb4113a60669da8d8556", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2815, "end_char_idx": 5679, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage example

### Single Media Mode

The model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):

```python
import av
import torch
import numpy as np
from transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor

def read_video_pyav(container, indices):
    '''
    Decode the video with PyAV decoder.
    Args:
        container (`av.container.input.InputContainer`): PyAV container.
        indices (`List[int]`): List of frame indices to decode.
    Returns:
        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
    '''
    frames = []
    container.seek(0)
    start_index = indices[0]
    end_index = indices[-1]
    for i, frame in enumerate(container.decode(video=0)):
        if i > end_index:
            break
        if i >= start_index and i in indices:
            frames.append(frame)
    return np.stack([x.to_ndarray(format="rgb24") for x in frames])

# Load the model in half-precision
model = VideoLlavaForConditionalGeneration.from_pretrained("LanguageBind/Video-LLaVA-7B-hf", torch_dtype=torch.float16, device_map="auto")
processor = VideoLlavaProcessor.from_pretrained("LanguageBind/Video-LLaVA-7B-hf")

# Load the video as an np.arrau, sampling uniformly 8 frames
video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
container = av.open(video_path)
total_frames = container.streams.video[0].frames
indices = np.arange(0, total_frames, total_frames / 8).astype(int)
video = read_video_pyav(container, indices)

# For better results, we recommend to prompt the model in the following format
prompt = "USER: <video>\nWhy is this funny? ASSISTANT:"
inputs = processor(text=prompt, videos=video, return_tensors="pt")

out = model.generate(**inputs, max_new_tokens=60)
processor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)
```

For multiple turns conversation change the prompt format to:

```bash
"USER: <video>\nWhat do you see in this video? ASSISTANT: A baby reading a book. USER: Why is the it funny? ASSISTANT:"
```

### Mixed Media Mode

The model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: 

```python
from PIL import Image
import requests

# Generate from image and video mixed inputs
# Load and image and write a new prompt
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
prompt = "USER: <image>\nHow many cats are there in the image? ASSISTANT: There are two cats. USER: <video>\nWhy is this video funny?


key: _node_content
string_value:
{"id_": "4245e9a0-d00b-48f3-a00d-c12e253bbe6e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98b923fb-63d8-4301-bab1-898028454305", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1c09b739643997ff775bdaa294e78304d2f40f3838f2d65cda4f4a9029aab41b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72686aed-d61f-4e94-aa52-a089dbd60eb1", "node_type": "1", "metadata": {}, "hash": "f0958f432101801dca225bcf8728ea85bc59940e6ac73b194fa35ee49e6ced23", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4574, "end_char_idx": 6941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
ASSISTANT:"
inputs = processor(text=prompt, videos=video, return_tensors="pt")

out = model.generate(**inputs, max_new_tokens=60)
processor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)
```

For multiple turns conversation change the prompt format to:

```bash
"USER: <video>\nWhat do you see in this video? ASSISTANT: A baby reading a book. USER: Why is the it funny? ASSISTANT:"
```

### Mixed Media Mode

The model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: 

```python
from PIL import Image
import requests

# Generate from image and video mixed inputs
# Load and image and write a new prompt
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
prompt = "USER: <image>\nHow many cats are there in the image? ASSISTANT: There are two cats. USER: <video>\nWhy is this video funny? ASSISTANT:"

inputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors="pt")

# Generate
generate_ids = model.generate(**inputs, max_length=50)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)

```

## Model optimization

### Quantization using Bitsandbytes for memory efficiency

The model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. his allows for efficient deployment on resource-constrained cases. 

First make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:


```python
from transformers import VideoLlavaForConditionalGeneration, BitsAndBytesConfig

# specify how to quantize the model
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = VideoLlavaForConditionalGeneration.from_pretrained("LanguageBind/Video-LLaVA-7B-hf", quantization_config=quantization_config, device_map="auto")
```


key: _node_content
string_value:
{"id_": "72686aed-d61f-4e94-aa52-a089dbd60eb1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4245e9a0-d00b-48f3-a00d-c12e253bbe6e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bdafe818e6c7388a2c943ff3741b91301dfba9d7f7b77914ff0deefb3a52c9e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5609, "end_char_idx": 8309, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
ASSISTANT: There are two cats. USER: <video>\nWhy is this video funny? ASSISTANT:"

inputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors="pt")

# Generate
generate_ids = model.generate(**inputs, max_length=50)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)

```

## Model optimization

### Quantization using Bitsandbytes for memory efficiency

The model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. his allows for efficient deployment on resource-constrained cases. 

First make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:


```python
from transformers import VideoLlavaForConditionalGeneration, BitsAndBytesConfig

# specify how to quantize the model
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = VideoLlavaForConditionalGeneration.from_pretrained("LanguageBind/Video-LLaVA-7B-hf", quantization_config=quantization_config, device_map="auto")
```


### Flash-Attention 2 to speed-up generation

Additionally, we can greatly speed-up model inference by using [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

Also, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.

To load and run a model using Flash Attention-2, simply add `attn_implementation="flash_attention_2"` when loading the model as follows:

```python
from transformers import VideoLlavaForConditionalGeneration

model = VideoLlavaForConditionalGeneration.from_pretrained(
    "LanguageBind/Video-LLaVA-7B-hf", 
    torch_dtype=torch.float16, 
    attn_implementation="flash_attention_2",
).to(0)
```


## VideoLlavaConfig

[[autodoc]] VideoLlavaConfig

## VideoLlavaImageProcessor

[[autodoc]] VideoLlavaImageProcessor

## VideoLlavaProcessor

[[autodoc]] VideoLlavaProcessor

## VideoLlavaForConditionalGeneration

[[autodoc]] VideoLlavaForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "80328af8-d89e-470f-9f46-75db5b5bfb93", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a2188600d847f2ffd6505a575b3783b5c119c24da1562b50a0fde6d51ecaa49a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cdfb2f72-be24-47b6-94ed-0829bb945d1a", "node_type": "1", "metadata": {}, "hash": "effd02f5291e1b8394a3f26c9c38728c1bce69dcc2f46c046297a53b881f3565", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3236, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MGP-STR

## Overview

The MGP-STR model was proposed in [Multi-Granularity Prediction for Scene Text Recognition](https://arxiv.org/abs/2209.03592) by Peng Wang, Cheng Da, and Cong Yao. MGP-STR is a conceptually **simple** yet **powerful** vision Scene Text Recognition (STR) model, which is built upon the [Vision Transformer (ViT)](vit). To integrate linguistic knowledge, Multi-Granularity Prediction (MGP) strategy is proposed to inject information from the language modality into the model in an implicit way.

The abstract from the paper is the following:

*Scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this challenging problem, numerous innovative methods have been successively proposed and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet powerful vision STR model, which is built upon ViT and outperforms previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely-used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. The resultant algorithm (termed MGP-STR) is able to push the performance envelop of STR to an even higher level. Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mgp_str_architecture.png"
alt="drawing" width="600"/>

<small> MGP-STR architecture. Taken from the <a href="https://arxiv.org/abs/2209.03592">original paper</a>. </small>

MGP-STR is trained on two synthetic datasets [MJSynth]((http://www.robots.ox.ac.uk/~vgg/data/text/)) (MJ) and [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE).
This model was contributed by [yuekun](https://huggingface.co/yuekun). The original code can be found [here](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).

## Inference example

[`MgpstrModel`] accepts images as input and generates three types of predictions, which represent textual information at different granularities.
The three types of predictions are fused to give the final prediction result.

The [`ViTImageProcessor`] class is responsible for preprocessing the input image and
[`MgpstrTokenizer`] decodes the generated character tokens to the target string. The
[`MgpstrProcessor`] wraps [`ViTImageProcessor`] and [`MgpstrTokenizer`]
into a single instance to both extract the input features and decode the predicted token ids.


key: _node_content
string_value:
{"id_": "cdfb2f72-be24-47b6-94ed-0829bb945d1a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a2188600d847f2ffd6505a575b3783b5c119c24da1562b50a0fde6d51ecaa49a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80328af8-d89e-470f-9f46-75db5b5bfb93", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "443b2ba9d34df9b05a0e51903aa0126c6492ba361ff389d771479c58e03e9ea0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1691, "end_char_idx": 4290, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mgp_str_architecture.png"
alt="drawing" width="600"/>

<small> MGP-STR architecture. Taken from the <a href="https://arxiv.org/abs/2209.03592">original paper</a>. </small>

MGP-STR is trained on two synthetic datasets [MJSynth]((http://www.robots.ox.ac.uk/~vgg/data/text/)) (MJ) and [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE).
This model was contributed by [yuekun](https://huggingface.co/yuekun). The original code can be found [here](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).

## Inference example

[`MgpstrModel`] accepts images as input and generates three types of predictions, which represent textual information at different granularities.
The three types of predictions are fused to give the final prediction result.

The [`ViTImageProcessor`] class is responsible for preprocessing the input image and
[`MgpstrTokenizer`] decodes the generated character tokens to the target string. The
[`MgpstrProcessor`] wraps [`ViTImageProcessor`] and [`MgpstrTokenizer`]
into a single instance to both extract the input features and decode the predicted token ids.

- Step-by-step Optical Character Recognition (OCR)

```py
>>> from transformers import MgpstrProcessor, MgpstrForSceneTextRecognition
>>> import requests
>>> from PIL import Image

>>> processor = MgpstrProcessor.from_pretrained('alibaba-damo/mgp-str-base')
>>> model = MgpstrForSceneTextRecognition.from_pretrained('alibaba-damo/mgp-str-base')

>>> # load image from the IIIT-5k dataset
>>> url = "https://i.postimg.cc/ZKwLg2Gw/367-14.png"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

>>> pixel_values = processor(images=image, return_tensors="pt").pixel_values
>>> outputs = model(pixel_values)

>>> generated_text = processor.batch_decode(outputs.logits)['generated_text']
```

## MgpstrConfig

[[autodoc]] MgpstrConfig

## MgpstrTokenizer

[[autodoc]] MgpstrTokenizer
    - save_vocabulary

## MgpstrProcessor

[[autodoc]] MgpstrProcessor
    - __call__
    - batch_decode

## MgpstrModel

[[autodoc]] MgpstrModel
    - forward

## MgpstrForSceneTextRecognition

[[autodoc]] MgpstrForSceneTextRecognition
    - forward


key: _node_content
string_value:
{"id_": "682d3aa8-4e0f-4cb7-9de4-da8a9693ced7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3d9e40b337b6e30e0b8e81ca60c733b8e0772d11332e013ad9bff337e9a34b5f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e24bd266-cd8b-43da-82e1-c9ea30baca85", "node_type": "1", "metadata": {}, "hash": "5efce74adbd19bf6f2effa81a4a3e3741e2f9e898c475ccbda2f5df62728a6f3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3237, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MEGA

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The MEGA model was proposed in [Mega: Moving Average Equipped Gated Attention](https://arxiv.org/abs/2209.10655) by Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.
MEGA proposes a new approach to self-attention with each encoder layer having a multi-headed exponential moving average in addition to a single head of standard dot-product attention, giving the attention mechanism
stronger positional biases. This allows MEGA to perform competitively to Transformers on standard benchmarks including LRA
while also having significantly fewer parameters. MEGA's compute efficiency allows it to scale to very long sequences, making it an
attractive option for long-document NLP tasks.

The abstract from the paper is the following:

 *The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of Mega that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that Mega achieves significant improvements over other sequence models, including variants of Transformers and recent state space models. *

This model was contributed by [mnaylor](https://huggingface.co/mnaylor).
The original code can be found [here](https://github.com/facebookresearch/mega).


## Usage tips

- MEGA can perform quite well with relatively few parameters. See Appendix D in the MEGA paper for examples of architectural specs which perform well in various settings. If using MEGA as a decoder, be sure to set `bidirectional=False` to avoid errors with default bidirectional.
- Mega-chunk is a variant of mega that reduces time and spaces complexity from quadratic to linear. Utilize chunking with MegaConfig.use_chunking and control chunk size with MegaConfig.chunk_size


## Implementation Notes

- The original implementation of MEGA had an inconsistent expectation of attention masks for padding and causal self-attention between the softmax attention and Laplace/squared ReLU method. This implementation addresses that inconsistency.
- The original implementation did not include token type embeddings; this implementation adds support for these, with the option controlled by MegaConfig.add_token_type_embeddings


key: _node_content
string_value:
{"id_": "e24bd266-cd8b-43da-82e1-c9ea30baca85", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3d9e40b337b6e30e0b8e81ca60c733b8e0772d11332e013ad9bff337e9a34b5f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "682d3aa8-4e0f-4cb7-9de4-da8a9693ced7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "92f42c82f66996199cebaa33c98cbc90080b4286809c143987886ff99ad87569", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2300, "end_char_idx": 3786, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips

- MEGA can perform quite well with relatively few parameters. See Appendix D in the MEGA paper for examples of architectural specs which perform well in various settings. If using MEGA as a decoder, be sure to set `bidirectional=False` to avoid errors with default bidirectional.
- Mega-chunk is a variant of mega that reduces time and spaces complexity from quadratic to linear. Utilize chunking with MegaConfig.use_chunking and control chunk size with MegaConfig.chunk_size


## Implementation Notes

- The original implementation of MEGA had an inconsistent expectation of attention masks for padding and causal self-attention between the softmax attention and Laplace/squared ReLU method. This implementation addresses that inconsistency.
- The original implementation did not include token type embeddings; this implementation adds support for these, with the option controlled by MegaConfig.add_token_type_embeddings


## MegaConfig

[[autodoc]] MegaConfig

## MegaModel

[[autodoc]] MegaModel
    - forward

## MegaForCausalLM

[[autodoc]] MegaForCausalLM
    - forward

## MegaForMaskedLM

[[autodoc]] MegaForMaskedLM
    - forward

## MegaForSequenceClassification

[[autodoc]] MegaForSequenceClassification
    - forward

## MegaForMultipleChoice

[[autodoc]] MegaForMultipleChoice
    - forward

## MegaForTokenClassification

[[autodoc]] MegaForTokenClassification
    - forward

## MegaForQuestionAnswering

[[autodoc]] MegaForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "8e69addb-b6ce-4b2f-8b51-3cbf22be844a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f5960213f2e4ed96b6a26343eac693534b3f6e36ce2557d7b3d07c8e88dc0f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd835cff-3ee7-4bcd-9526-a0b9866cfaab", "node_type": "1", "metadata": {}, "hash": "e6e6b44f9559abe8059e29715d25697fef0aa9ce310414362715890a42d84629", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2841, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CodeGen

## Overview

The CodeGen model was proposed in [A Conversational Paradigm for Program Synthesis](https://arxiv.org/abs/2203.13474) by Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.

CodeGen is an autoregressive language model for program synthesis trained sequentially on [The Pile](https://pile.eleuther.ai/), BigQuery, and BigPython.

The abstract from the paper is the following:

*Program synthesis strives to generate a computer program as a solution to a given problem specification. We propose a conversational program synthesis approach via large language models, which addresses the challenges of searching over a vast program space and user intent specification faced in prior approaches. Our new approach casts the process of writing a specification and program as a multi-turn conversation between a user and a system. It treats program synthesis as a sequence prediction problem, in which the specification is expressed in natural language and the desired program is conditionally sampled. We train a family of large language models, called CodeGen, on natural language and programming language data. With weak supervision in the data and the scaling up of data size and model size, conversational capacities emerge from the simple autoregressive language modeling. To study the model behavior on conversational program synthesis, we develop a multi-turn programming benchmark (MTPB), where solving each problem requires multi-step synthesis via multi-turn conversation between the user and the model. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm. In addition, our model CodeGen (with up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the HumanEval benchmark. We make the training library JaxFormer including checkpoints available as open source contribution: [this https URL](https://github.com/salesforce/codegen).* 

This model was contributed by [Hiroaki Hayashi](https://huggingface.co/rooa).
The original code can be found [here](https://github.com/salesforce/codegen).

## Checkpoint Naming

* CodeGen model [checkpoints](https://huggingface.co/models?other=codegen) are available on different pre-training data with variable sizes.
* The format is: `Salesforce/codegen-{size}-{data}`, where
  * `size`: `350M`, `2B`, `6B`, `16B`
  * `data`: 
    * `nl`: Pre-trained on the Pile
    * `multi`: Initialized with `nl`, then further pre-trained on multiple programming languages data
    * `mono`: Initialized with `multi`, then further pre-trained on Python data
* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python.


key: _node_content
string_value:
{"id_": "cd835cff-3ee7-4bcd-9526-a0b9866cfaab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8f5960213f2e4ed96b6a26343eac693534b3f6e36ce2557d7b3d07c8e88dc0f2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e69addb-b6ce-4b2f-8b51-3cbf22be844a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "67b6a0662f7aa5517fa47a59ea6fb98bd4dde385fbf25541d32b833e1f8b11b9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1076, "end_char_idx": 3755, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We train a family of large language models, called CodeGen, on natural language and programming language data. With weak supervision in the data and the scaling up of data size and model size, conversational capacities emerge from the simple autoregressive language modeling. To study the model behavior on conversational program synthesis, we develop a multi-turn programming benchmark (MTPB), where solving each problem requires multi-step synthesis via multi-turn conversation between the user and the model. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm. In addition, our model CodeGen (with up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the HumanEval benchmark. We make the training library JaxFormer including checkpoints available as open source contribution: [this https URL](https://github.com/salesforce/codegen).* 

This model was contributed by [Hiroaki Hayashi](https://huggingface.co/rooa).
The original code can be found [here](https://github.com/salesforce/codegen).

## Checkpoint Naming

* CodeGen model [checkpoints](https://huggingface.co/models?other=codegen) are available on different pre-training data with variable sizes.
* The format is: `Salesforce/codegen-{size}-{data}`, where
  * `size`: `350M`, `2B`, `6B`, `16B`
  * `data`: 
    * `nl`: Pre-trained on the Pile
    * `multi`: Initialized with `nl`, then further pre-trained on multiple programming languages data
    * `mono`: Initialized with `multi`, then further pre-trained on Python data
* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python.

## Usage example

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> checkpoint = "Salesforce/codegen-350M-mono"
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)

>>> text = "def hello_world():"

>>> completion = model.generate(**tokenizer(text, return_tensors="pt"))

>>> print(tokenizer.decode(completion[0]))
def hello_world():
    print("Hello World")

hello_world()
```

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## CodeGenConfig

[[autodoc]] CodeGenConfig
    - all

## CodeGenTokenizer

[[autodoc]] CodeGenTokenizer
    - create_token_type_ids_from_sequences
    - save_vocabulary

## CodeGenTokenizerFast

[[autodoc]] CodeGenTokenizerFast

## CodeGenModel

[[autodoc]] CodeGenModel
    - forward

## CodeGenForCausalLM

[[autodoc]] CodeGenForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "4451d02c-d07a-422e-ae2d-c92c50ee87a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "eb400873-3e05-4025-8800-fcde494f31de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc", "node_type": "1", "metadata": {}, "hash": "06a49eae299cbe977a8db5498e8706b78b041dfa1ca26991b1aa6c6b1be911e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2844, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Cohere

## Overview

The Cohere Command-R model was proposed in the blogpost [Command-R: Retrieval Augmented Generation at Production Scale](https://txt.cohere.com/command-r/) by the Cohere Team.

The abstract from the paper is the following:

*Command-R is a scalable generative model targeting RAG and Tool Use to enable production-scale AI for enterprise. Today, we are introducing Command-R, a new LLM aimed at large-scale production workloads. Command-R targets the emerging “scalable” category of models that balance high efficiency with strong accuracy, enabling companies to move beyond proof of concept, and into production.*

*Command-R is a generative model optimized for long context tasks such as retrieval augmented generation (RAG) and using external APIs and tools. It is designed to work in concert with our industry-leading Embed and Rerank models to provide best-in-class integration for RAG applications and excel at enterprise use cases. As a model built for companies to implement at scale, Command-R boasts:
- Strong accuracy on RAG and Tool Use
- Low latency, and high throughput
- Longer 128k context and lower pricing
- Strong capabilities across 10 key languages
- Model weights available on HuggingFace for research and evaluation

Checkout model checkpoints [here](https://huggingface.co/CohereForAI/c4ai-command-r-v01).
This model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet Üstün](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox).

## Usage tips

<Tip warning={true}>

The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be
used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. 

The `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. 

Training the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.

</Tip>
The model and tokenizer can be loaded via:

```python
# pip install transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Format message with the command-r chat template
messages = [{"role": "user", "content": "Hello, how are you?"}]


key: _node_content
string_value:
{"id_": "6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "eb400873-3e05-4025-8800-fcde494f31de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4451d02c-d07a-422e-ae2d-c92c50ee87a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "56b743109eeacbf2ea86d7fce5b32291e2c53ce7f5ef0d2cd785b28cbf5095eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0bcbbe2-ecf0-4c56-9844-344657e6a0df", "node_type": "1", "metadata": {}, "hash": "98896c33887a60da0521ff2a10d42500790e0aa73423af8aeeb260cdaef48621", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1352, "end_char_idx": 4038, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet Üstün](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox).

## Usage tips

<Tip warning={true}>

The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be
used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. 

The `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. 

Training the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.

</Tip>
The model and tokenizer can be loaded via:

```python
# pip install transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Format message with the command-r chat template
messages = [{"role": "user", "content": "Hello, how are you?"}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
```

- When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.


## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "c0bcbbe2-ecf0-4c56-9844-344657e6a0df", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "eb400873-3e05-4025-8800-fcde494f31de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "15e43647aca6fbfc69b3b819263f5d911defb4db31d1cf56e7d2e0fbcada12a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92e559ab-f011-42ee-a39a-352ddf20e8c8", "node_type": "1", "metadata": {}, "hash": "15e6216aa2d9b487337d0d692fd8cf2bad2093eadcd639f533de92c2e2180395", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2429, "end_char_idx": 5500, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
The model and tokenizer can be loaded via:

```python
# pip install transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Format message with the command-r chat template
messages = [{"role": "user", "content": "Hello, how are you?"}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
```

- When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.


## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


<PipelineTag pipeline="text-generation"/>

Loading FP16 model
```python
# pip install transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Format message with the command-r chat template
messages = [{"role": "user", "content": "Hello, how are you?"}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
```

Loading bitsnbytes 4bit quantized model
```python
# pip install transformers bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(load_in_4bit=True)

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
```


key: _node_content
string_value:
{"id_": "92e559ab-f011-42ee-a39a-352ddf20e8c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "eb400873-3e05-4025-8800-fcde494f31de", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c0bcbbe2-ecf0-4c56-9844-344657e6a0df", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "74dd728b49c02d8afa11a7807a6b9a69a9cf8d52600a6cdf83b4481e76b82940", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4041, "end_char_idx": 5886, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<PipelineTag pipeline="text-generation"/>

Loading FP16 model
```python
# pip install transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Format message with the command-r chat template
messages = [{"role": "user", "content": "Hello, how are you?"}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
```

Loading bitsnbytes 4bit quantized model
```python
# pip install transformers bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(load_in_4bit=True)

model_id = "CohereForAI/c4ai-command-r-v01"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)
```


## CohereConfig

[[autodoc]] CohereConfig

## CohereTokenizerFast

[[autodoc]] CohereTokenizerFast
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - update_post_processor
    - save_vocabulary

## CohereModel

[[autodoc]] CohereModel
    - forward


## CohereForCausalLM

[[autodoc]] CohereForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "2ad42a43-3169-4abc-a60d-fb6ac3118f56", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d428f6f790b0762383386442074bc8ee4074011b4d0dfe60e8dea6e67cfbd4f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d5e0a73-a831-4fc7-9e5d-291e7d98f67c", "node_type": "1", "metadata": {}, "hash": "d3d9af46590fc64f156313f524aa4e75ad480d6763b22a922cfd7b10ba3e2589", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2353, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ProphetNet

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=prophetnet">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-prophetnet-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/prophetnet-large-uncased">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The ProphetNet model was proposed in [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
Zhang, Ming Zhou on 13 Jan, 2020.

ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of just
the next token.

The abstract from the paper is the following:

*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel
self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent
overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
state-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*

The Authors' code can be found [here](https://github.com/microsoft/ProphetNet).

## Usage tips

- ProphetNet is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- The model architecture is based on the original Transformer, but replaces the “standard” self-attention mechanism in the decoder by a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism.


key: _node_content
string_value:
{"id_": "6d5e0a73-a831-4fc7-9e5d-291e7d98f67c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d428f6f790b0762383386442074bc8ee4074011b4d0dfe60e8dea6e67cfbd4f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ad42a43-3169-4abc-a60d-fb6ac3118f56", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d3c38a7b35cea025c6567465e85fa303b023c640c6676022bedd7f272723f655", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 673, "end_char_idx": 3365, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of just
the next token.

The abstract from the paper is the following:

*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel
self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent
overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
state-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*

The Authors' code can be found [here](https://github.com/microsoft/ProphetNet).

## Usage tips

- ProphetNet is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- The model architecture is based on the original Transformer, but replaces the “standard” self-attention mechanism in the decoder by a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism.

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## ProphetNetConfig

[[autodoc]] ProphetNetConfig

## ProphetNetTokenizer

[[autodoc]] ProphetNetTokenizer

## ProphetNet specific outputs

[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqLMOutput

[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqModelOutput

[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetDecoderModelOutput

[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetDecoderLMOutput

## ProphetNetModel

[[autodoc]] ProphetNetModel
    - forward

## ProphetNetEncoder

[[autodoc]] ProphetNetEncoder
    - forward

## ProphetNetDecoder

[[autodoc]] ProphetNetDecoder
    - forward

## ProphetNetForConditionalGeneration

[[autodoc]] ProphetNetForConditionalGeneration
    - forward

## ProphetNetForCausalLM

[[autodoc]] ProphetNetForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "a8557bde-3dec-4d16-ae21-d1a0c9ad887b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a19e8fb-f4a9-4e18-a39e-18044058e498", "node_type": "1", "metadata": {}, "hash": "4ea3143e15481c06ef3dbe0ea85abe948696006a18018d72b934560e364f4f3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2773, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPT-NeoX

## Overview

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will
be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge,
the largest dense autoregressive model that has publicly available weights at the time of submission. In this work,
we describe GPT-NeoX-20B's architecture and training and evaluate its performance on a range of language-understanding,
mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and
gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source
the training and evaluation code, as well as the model weights, at [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).

Development of the model was led by Sid Black, Stella Biderman and Eric Hallahan, and the model was trained with
generous the support of [CoreWeave](https://www.coreweave.com/).

GPT-NeoX-20B was trained with fp16, thus it is recommended to initialize the model as follows:

```python
model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b").half().cuda()
```

GPT-NeoX-20B also has a different tokenizer from the one used in GPT-J-6B and GPT-Neo. The new tokenizer allocates
additional tokens to whitespace characters, making the model more suitable for certain tasks like code generation.

## Usage example

The `generate()` method can be used to generate text using GPT Neo model.

```python
>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast

>>> model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b")
>>> tokenizer = GPTNeoXTokenizerFast.from_pretrained("EleutherAI/gpt-neox-20b")

>>> prompt = "GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI."

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

## Using Flash Attention 2

Flash Attention 2 is an faster, optimized version of the model.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).


key: _node_content
string_value:
{"id_": "8a19e8fb-f4a9-4e18-a39e-18044058e498", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8557bde-3dec-4d16-ae21-d1a0c9ad887b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5a84f17abcf55e69a23b745e197860cfad8a84f99f1021a35aeb8c97adb9f086", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "583f6358-24ee-4f5a-83b8-a92e44747723", "node_type": "1", "metadata": {}, "hash": "66d0a5e20132381cda993924da98df5706c9cfe258c1253a834c9698cb749e98", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1341, "end_char_idx": 4350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The new tokenizer allocates
additional tokens to whitespace characters, making the model more suitable for certain tasks like code generation.

## Usage example

The `generate()` method can be used to generate text using GPT Neo model.

```python
>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast

>>> model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b")
>>> tokenizer = GPTNeoXTokenizerFast.from_pretrained("EleutherAI/gpt-neox-20b")

>>> prompt = "GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI."

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

## Using Flash Attention 2

Flash Attention 2 is an faster, optimized version of the model.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast

model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
...
```


### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `stockmark/gpt-neox-japanese-1.4b` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/gpt-neox-1.8b-speedup.jpg">
</div>


## Using Scaled Dot Product Attention (SDPA)
PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function
encompasses several implementations that can be applied depending on the inputs and the hardware in use.


key: _node_content
string_value:
{"id_": "583f6358-24ee-4f5a-83b8-a92e44747723", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a19e8fb-f4a9-4e18-a39e-18044058e498", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "35249976ad59ca00ddf95e2fc4a411440742d598ff69ad9d83e1862797a7d6dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e8c0589-e474-4444-8550-fe374ce7e9db", "node_type": "1", "metadata": {}, "hash": "aed132cddf7d762190785d510ade4241d0f314cf5e4b6edfeac5f95304ace185", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2775, "end_char_idx": 5792, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast

model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
...
```


### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `stockmark/gpt-neox-japanese-1.4b` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/gpt-neox-1.8b-speedup.jpg">
</div>


## Using Scaled Dot Product Attention (SDPA)
PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```python
from transformers import GPTNeoXForCausalLM
model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b", torch_dtype=torch.float16, attn_implementation="sdpa")
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with
[pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped), we saw the
following speedups during training and inference.

### Training
| Batch size |    Seq len | Time per batch (Eager - s) |    Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |
|-----------:|-----------:|---------------------------:|-----------------------------:|------------:|--------------------:|-------------------:|------------------:|
|          1 |        128 |                      0.024 |                        0.


key: _node_content
string_value:
{"id_": "5e8c0589-e474-4444-8550-fe374ce7e9db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "583f6358-24ee-4f5a-83b8-a92e44747723", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "06b4a2dbc99446cb67e87c257cdecb853b25c83a9e08d76a72d818e2c5a07f22", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "08d249c5-84b6-4758-8092-d86c16a2e54e", "node_type": "1", "metadata": {}, "hash": "d5aa62e8f963b5cf99315654a67f2a32fef652371576c01c0b9b70b0644cdd45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4351, "end_char_idx": 7126, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See the
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```python
from transformers import GPTNeoXForCausalLM
model = GPTNeoXForCausalLM.from_pretrained("EleutherAI/gpt-neox-20b", torch_dtype=torch.float16, attn_implementation="sdpa")
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with
[pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped), we saw the
following speedups during training and inference.

### Training
| Batch size |    Seq len | Time per batch (Eager - s) |    Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |
|-----------:|-----------:|---------------------------:|-----------------------------:|------------:|--------------------:|-------------------:|------------------:|
|          1 |        128 |                      0.024 |                        0.019 |      28.945 |             1789.95 |            1789.95 |                 0 |
|          1 |        256 |                      0.039 |                        0.031 |       23.18 |             1845.83 |            1844.84 |             0.053 |
|          1 |        512 |                       0.08 |                        0.055 |      45.524 |             2278.38 |            1953.76 |            16.615 |
|          1 |       1024 |                       0.19 |                        0.102 |      86.777 |             4772.36 |            2408.35 |            98.159 |
|          1 |       2048 |                      0.565 |                        0.204 |     177.098 |             13484.1 |            3882.01 |           247.348 |
|          2 |        128 |                      0.037 |                        0.032 |      15.121 |             1843.86 |            1844.78 |             -0.05 |
|          2 |        256 |                      0.067 |                        0.055 |      21.706 |             1999.72 |            1951.67 |             2.462 |
|          2 |        512 |                      0.144 |                        0.096 |      50.046 |             3613.16 |            2406.77 |            50.125 |
|          2 |       1024 |                      0.366 |                        0.193 |      89.


key: _node_content
string_value:
{"id_": "08d249c5-84b6-4758-8092-d86c16a2e54e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e8c0589-e474-4444-8550-fe374ce7e9db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f8bce7e3b86e723d7b2054b806ec6ef40b77144ba092d1b7c3c2be8d25d9e57d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a23ac3b4-f83b-45ae-9ffc-8569806b4a45", "node_type": "1", "metadata": {}, "hash": "d4a7402e1aa1033db1de0865a8b3e08b68165389a06bd26544496bc1eead35b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5761, "end_char_idx": 8642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
024 |                        0.019 |      28.945 |             1789.95 |            1789.95 |                 0 |
|          1 |        256 |                      0.039 |                        0.031 |       23.18 |             1845.83 |            1844.84 |             0.053 |
|          1 |        512 |                       0.08 |                        0.055 |      45.524 |             2278.38 |            1953.76 |            16.615 |
|          1 |       1024 |                       0.19 |                        0.102 |      86.777 |             4772.36 |            2408.35 |            98.159 |
|          1 |       2048 |                      0.565 |                        0.204 |     177.098 |             13484.1 |            3882.01 |           247.348 |
|          2 |        128 |                      0.037 |                        0.032 |      15.121 |             1843.86 |            1844.78 |             -0.05 |
|          2 |        256 |                      0.067 |                        0.055 |      21.706 |             1999.72 |            1951.67 |             2.462 |
|          2 |        512 |                      0.144 |                        0.096 |      50.046 |             3613.16 |            2406.77 |            50.125 |
|          2 |       1024 |                      0.366 |                        0.193 |      89.666 |             8707.55 |            3878.86 |           124.487 |
|          2 |       2048 |                        OOM |                        0.379 |           / |                 OOM |            6825.13 | SDPA does not OOM |
|          4 |        128 |                       0.06 |                        0.054 |      11.539 |              1947.6 |            1952.06 |            -0.228 |
|          4 |        256 |                      0.119 |                        0.093 |      28.072 |             3008.39 |            2405.99 |            25.038 |
|          4 |        512 |                      0.275 |                        0.187 |      47.145 |             6290.58 |            3877.29 |            62.242 |
|          4 |       1024 |                        OOM |                         0.36 |           / |                 OOM |            6821.98 | SDPA does not OOM |
|          4 |       2048 |                        OOM |                        0.731 |           / |                 OOM |            12705.1 | SDPA does not OOM |

### Inference
|    Batch size |      Seq len |    Per token latency Eager (ms) |    Per token latency SDPA (ms) |    Speedup (%) |    Mem Eager (MB) |   Mem SDPA (MB) |    Mem saved (%) |
|--------------:|-------------:|--------------------------------:|-------------------------------:|---------------:|------------------:|----------------:|-----------------:|
|             1 |          128 |                           6.569 |                          5.


key: _node_content
string_value:
{"id_": "a23ac3b4-f83b-45ae-9ffc-8569806b4a45", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "08d249c5-84b6-4758-8092-d86c16a2e54e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "737f88890d00fbf5d457c75fe4f3ada7d9647d73c3159fa3f6dd3e4e77fdceab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b996b549-3348-4c3e-89f1-751497e56450", "node_type": "1", "metadata": {}, "hash": "25c80eb907589a49695a45bc851b5bb9e2419288568874be07af4329f1542df3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7112, "end_char_idx": 10051, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
193 |      89.666 |             8707.55 |            3878.86 |           124.487 |
|          2 |       2048 |                        OOM |                        0.379 |           / |                 OOM |            6825.13 | SDPA does not OOM |
|          4 |        128 |                       0.06 |                        0.054 |      11.539 |              1947.6 |            1952.06 |            -0.228 |
|          4 |        256 |                      0.119 |                        0.093 |      28.072 |             3008.39 |            2405.99 |            25.038 |
|          4 |        512 |                      0.275 |                        0.187 |      47.145 |             6290.58 |            3877.29 |            62.242 |
|          4 |       1024 |                        OOM |                         0.36 |           / |                 OOM |            6821.98 | SDPA does not OOM |
|          4 |       2048 |                        OOM |                        0.731 |           / |                 OOM |            12705.1 | SDPA does not OOM |

### Inference
|    Batch size |      Seq len |    Per token latency Eager (ms) |    Per token latency SDPA (ms) |    Speedup (%) |    Mem Eager (MB) |   Mem SDPA (MB) |    Mem saved (%) |
|--------------:|-------------:|--------------------------------:|-------------------------------:|---------------:|------------------:|----------------:|-----------------:|
|             1 |          128 |                           6.569 |                          5.858 |          12.14 |           974.831 |         974.826 |                0 |
|             1 |          256 |                           7.009 |                          5.863 |         19.542 |           1029.01 |         1028.08 |             0.09 |
|             1 |          512 |                           7.157 |                          5.965 |         19.983 |           1137.54 |         1137.52 |            0.001 |
|             1 |         1024 |                           7.523 |                          6.506 |         15.637 |            1329.3 |         1329.26 |            0.003 |
|             1 |         2048 |                           9.271 |                          9.205 |          0.713 |           1752.47 |         1734.51 |            1.036 |
|             2 |          128 |                           7.239 |                          5.959 |         21.493 |            1044.8 |         1028.37 |            1.597 |
|             2 |          256 |                           7.228 |                          6.036 |         19.757 |           1167.32 |         1137.73 |            2.601 |
|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |
|             2 |         1024 |                           8.916 |                          8.632 |          3.


key: _node_content
string_value:
{"id_": "b996b549-3348-4c3e-89f1-751497e56450", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a23ac3b4-f83b-45ae-9ffc-8569806b4a45", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "717ab8f06a836e7b1f96ef1060819369bac0c8b643c9b0574c628fb274897a66", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "27d68e86-e731-4558-9451-916c04fe8949", "node_type": "1", "metadata": {}, "hash": "70c6c0b621f7905838cfa95f82bb2e260928033966bce8fe6c72fcfac3e9a074", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8609, "end_char_idx": 11157, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
569 |                          5.858 |          12.14 |           974.831 |         974.826 |                0 |
|             1 |          256 |                           7.009 |                          5.863 |         19.542 |           1029.01 |         1028.08 |             0.09 |
|             1 |          512 |                           7.157 |                          5.965 |         19.983 |           1137.54 |         1137.52 |            0.001 |
|             1 |         1024 |                           7.523 |                          6.506 |         15.637 |            1329.3 |         1329.26 |            0.003 |
|             1 |         2048 |                           9.271 |                          9.205 |          0.713 |           1752.47 |         1734.51 |            1.036 |
|             2 |          128 |                           7.239 |                          5.959 |         21.493 |            1044.8 |         1028.37 |            1.597 |
|             2 |          256 |                           7.228 |                          6.036 |         19.757 |           1167.32 |         1137.73 |            2.601 |
|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |
|             2 |         1024 |                           8.916 |                          8.632 |          3.291 |           1752.56 |         1734.62 |            1.034 |
|             2 |         2048 |                          12.628 |                         12.606 |          0.181 |           2558.72 |          2545.8 |            0.508 |
|             4 |          128 |                           7.278 |                          6.046 |         20.373 |           1168.41 |         1137.79 |            2.691 |
|             4 |          256 |                           7.614 |                          6.588 |         15.574 |            1353.1 |         1329.79 |            1.753 |
|             4 |          512 |                           8.798 |                          8.144 |          8.028 |           1752.76 |         1734.85 |            1.032 |
|             4 |         1024 |                          11.765 |                         11.303 |           4.09 |           2558.96 |         2546.04 |            0.508 |
|             4 |         2048 |                          19.568 |                         17.735 |          10.33 |            4175.5 |         4165.26 |            0.246 |


key: _node_content
string_value:
{"id_": "27d68e86-e731-4558-9451-916c04fe8949", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b996b549-3348-4c3e-89f1-751497e56450", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f11d9cb6fdf08041c781fe8a188651e75124fbd1e67c7e64f2e692efad247639", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9724, "end_char_idx": 11746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
32 |         1137.73 |            2.601 |
|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |
|             2 |         1024 |                           8.916 |                          8.632 |          3.291 |           1752.56 |         1734.62 |            1.034 |
|             2 |         2048 |                          12.628 |                         12.606 |          0.181 |           2558.72 |          2545.8 |            0.508 |
|             4 |          128 |                           7.278 |                          6.046 |         20.373 |           1168.41 |         1137.79 |            2.691 |
|             4 |          256 |                           7.614 |                          6.588 |         15.574 |            1353.1 |         1329.79 |            1.753 |
|             4 |          512 |                           8.798 |                          8.144 |          8.028 |           1752.76 |         1734.85 |            1.032 |
|             4 |         1024 |                          11.765 |                         11.303 |           4.09 |           2558.96 |         2546.04 |            0.508 |
|             4 |         2048 |                          19.568 |                         17.735 |          10.33 |            4175.5 |         4165.26 |            0.246 |


## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## GPTNeoXConfig

[[autodoc]] GPTNeoXConfig

## GPTNeoXTokenizerFast

[[autodoc]] GPTNeoXTokenizerFast

## GPTNeoXModel

[[autodoc]] GPTNeoXModel
    - forward

## GPTNeoXForCausalLM

[[autodoc]] GPTNeoXForCausalLM
    - forward

## GPTNeoXForQuestionAnswering

[[autodoc]] GPTNeoXForQuestionAnswering
    - forward

## GPTNeoXForSequenceClassification

[[autodoc]] GPTNeoXForSequenceClassification
    - forward

## GPTNeoXForTokenClassification

[[autodoc]] GPTNeoXForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "adeb55c5-134c-423f-8ea6-61d25381be36", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fsmt/", "title": "FSMT", "tokens": 619, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0e7e2bc-6e51-473b-ae86-5076a6bccdda", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fsmt/", "title": "FSMT", "tokens": 619, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c1dd6154a2721cf765d47aef416534cb2446ee17b54fed9ff9f9fd0b3569356", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1964, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FSMT

## Overview

FSMT (FairSeq MachineTranslation) models were introduced in [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616) by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.

The abstract of the paper is the following:

*This paper describes Facebook FAIR's submission to the WMT19 shared news translation task. We participate in two
language pairs and four language directions, English <-> German and English <-> Russian. Following our submission from
last year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling
toolkit which rely on sampled back-translations. This year we experiment with different bitext data filtering schemes,
as well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific
data, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the
human evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations.
This system improves upon our WMT'18 submission by 4.5 BLEU points.*

This model was contributed by [stas](https://huggingface.co/stas). The original code can be found
[here](https://github.com/pytorch/fairseq/tree/master/examples/wmt19).

## Implementation Notes

- FSMT uses source and target vocabulary pairs that aren't combined into one. It doesn't share embeddings tokens
  either. Its tokenizer is very similar to [`XLMTokenizer`] and the main model is derived from
  [`BartModel`].


## FSMTConfig

[[autodoc]] FSMTConfig

## FSMTTokenizer

[[autodoc]] FSMTTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## FSMTModel

[[autodoc]] FSMTModel
    - forward

## FSMTForConditionalGeneration

[[autodoc]] FSMTForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "2ccc31b0-7d91-4041-b4c4-a753b4a0227e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5ab0e68bf60af7098408bedf6b9e850e0d1f33e9dfc6422e4c145f247b70b0f0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f5cbaf6-8edd-4364-a9ca-864de9a279c4", "node_type": "1", "metadata": {}, "hash": "40cdcd1a879c8d17fb25945fd2d6ea4018dd8551dbf02b230be59f85def6497e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2635, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# mT5

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=mt5">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-mt5-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/mt5-small-finetuned-arxiv-cs-finetuned-arxiv-cs-full">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The mT5 model was presented in [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934) by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya
Siddhant, Aditya Barua, Colin Raffel.

The abstract from the paper is the following:

*The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain
state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a
multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail
the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual
benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a
generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model
checkpoints used in this work are publicly available.*

Note: mT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.
Therefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.
Since mT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task
fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.

Google has released the following variants:

- [google/mt5-small](https://huggingface.co/google/mt5-small)

- [google/mt5-base](https://huggingface.co/google/mt5-base)

- [google/mt5-large](https://huggingface.co/google/mt5-large)

- [google/mt5-xl](https://huggingface.co/google/mt5-xl)

- [google/mt5-xxl](https://huggingface.co/google/mt5-xxl).

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://github.com/google-research/multilingual-t5).

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## MT5Config

[[autodoc]] MT5Config

## MT5Tokenizer

[[autodoc]] MT5Tokenizer

See [`T5Tokenizer`] for all details.


key: _node_content
string_value:
{"id_": "5f5cbaf6-8edd-4364-a9ca-864de9a279c4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5ab0e68bf60af7098408bedf6b9e850e0d1f33e9dfc6422e4c145f247b70b0f0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ccc31b0-7d91-4041-b4c4-a753b4a0227e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a25e6d309e407335169d5bb8b53b13f75b8b7252ecde8b90bd7af39d01ee4330", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2638, "end_char_idx": 3523, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## MT5TokenizerFast

[[autodoc]] MT5TokenizerFast

See [`T5TokenizerFast`] for all details.

<frameworkcontent>
<pt>

## MT5Model

[[autodoc]] MT5Model

## MT5ForConditionalGeneration

[[autodoc]] MT5ForConditionalGeneration

## MT5EncoderModel

[[autodoc]] MT5EncoderModel

## MT5ForSequenceClassification

[[autodoc]] MT5ForSequenceClassification

## MT5ForTokenClassification

[[autodoc]] MT5ForTokenClassification

## MT5ForQuestionAnswering

[[autodoc]] MT5ForQuestionAnswering

</pt>
<tf>

## TFMT5Model

[[autodoc]] TFMT5Model

## TFMT5ForConditionalGeneration

[[autodoc]] TFMT5ForConditionalGeneration

## TFMT5EncoderModel

[[autodoc]] TFMT5EncoderModel

</tf>
<jax>

## FlaxMT5Model

[[autodoc]] FlaxMT5Model

## FlaxMT5ForConditionalGeneration

[[autodoc]] FlaxMT5ForConditionalGeneration

## FlaxMT5EncoderModel

[[autodoc]] FlaxMT5EncoderModel

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "20a709a5-f830-4284-a68c-14c4be27f734", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ad480bf7077c626676bf59ca40b5c1339a4a4ace942c2e647160abf72f8be1c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "54818bc1-466f-4554-a6d0-089ac525d335", "node_type": "1", "metadata": {}, "hash": "61b5e04bc01bc1c45f0e07a9e1b382e9366f6559a8cf6587865cdd5a40b13285", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Hiera

## Overview

Hiera was proposed in [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://arxiv.org/abs/2306.00989) by Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer

The paper introduces "Hiera," a hierarchical Vision Transformer that simplifies the architecture of modern hierarchical vision transformers by removing unnecessary components without compromising on accuracy or efficiency. Unlike traditional transformers that add complex vision-specific components to improve supervised classification performance, Hiera demonstrates that such additions, often termed "bells-and-whistles," are not essential for high accuracy. By leveraging a strong visual pretext task (MAE) for pretraining, Hiera retains simplicity and achieves superior accuracy and speed both in inference and training across various image and video recognition tasks. The approach suggests that spatial biases required for vision tasks can be effectively learned through proper pretraining, eliminating the need for added architectural complexity. 

The abstract from the paper is the following:

*Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/hiera_overview.png"
alt="drawing" width="600"/>

<small> Hiera architecture. Taken from the <a href="https://arxiv.org/abs/2306.00989">original paper.</a> </small>

This model was a joint contibution by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [namangarg110](https://huggingface.co/namangarg110). The original code can be found [here] (https://github.com/facebookresearch/hiera).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Hiera. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="image-classification"/>

- [`HieraForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).


key: _node_content
string_value:
{"id_": "54818bc1-466f-4554-a6d0-089ac525d335", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ad480bf7077c626676bf59ca40b5c1339a4a4ace942c2e647160abf72f8be1c2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "20a709a5-f830-4284-a68c-14c4be27f734", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9f7f2d5698ced7d7378179e513844690570841e6062814e55ada48011a27350f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1650, "end_char_idx": 3716, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/hiera_overview.png"
alt="drawing" width="600"/>

<small> Hiera architecture. Taken from the <a href="https://arxiv.org/abs/2306.00989">original paper.</a> </small>

This model was a joint contibution by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [namangarg110](https://huggingface.co/namangarg110). The original code can be found [here] (https://github.com/facebookresearch/hiera).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Hiera. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="image-classification"/>

- [`HieraForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

## HieraConfig

[[autodoc]] HieraConfig

## HieraModel

[[autodoc]] HieraModel
    - forward

## HieraForPreTraining

[[autodoc]] HieraForPreTraining
    - forward
  
## HieraForImageClassification

[[autodoc]] HieraForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "2b1925e0-5cb7-4e65-9f96-1a4689e95a80", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d04419119925e60feed4a635622f6e561c5d982934b76364e8f7ade284666b36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a57adba6-fc57-4a49-a739-02b530dab15d", "node_type": "1", "metadata": {}, "hash": "3540d043a01b0486307106a78e2201e6ac349b060dd8d39f363f229f14d35781", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2185, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ConvBERT

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=convbert">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-convbert-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/conv-bert-base">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The ConvBERT model was proposed in [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://arxiv.org/abs/2008.02496) by Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng
Yan.

The abstract from the paper is the following:

*Pre-trained language models like BERT and its variants have recently achieved impressive performance in various
natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers
large memory footprint and computation cost. Although all its attention heads query on the whole input sequence for
generating the attention map from a global perspective, we observe some heads only need to learn local dependencies,
which means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to
replace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the
rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context
learning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that
ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and
fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while
using less than 1/4 training cost. Code and pre-trained models will be released.*

This model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found
here: https://github.com/yitu-opensource/ConvBert

## Usage tips

ConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert).


key: _node_content
string_value:
{"id_": "a57adba6-fc57-4a49-a739-02b530dab15d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d04419119925e60feed4a635622f6e561c5d982934b76364e8f7ade284666b36", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b1925e0-5cb7-4e65-9f96-1a4689e95a80", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4965c64026a5226bae1338391b9af032eaa0447f0cf5a5f44520a10f88dfce3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 608, "end_char_idx": 3884, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract from the paper is the following:

*Pre-trained language models like BERT and its variants have recently achieved impressive performance in various
natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers
large memory footprint and computation cost. Although all its attention heads query on the whole input sequence for
generating the attention map from a global perspective, we observe some heads only need to learn local dependencies,
which means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to
replace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the
rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context
learning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that
ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and
fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while
using less than 1/4 training cost. Code and pre-trained models will be released.*

This model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found
here: https://github.com/yitu-opensource/ConvBert

## Usage tips

ConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## ConvBertConfig

[[autodoc]] ConvBertConfig

## ConvBertTokenizer

[[autodoc]] ConvBertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## ConvBertTokenizerFast

[[autodoc]] ConvBertTokenizerFast

<frameworkcontent>
<pt>

## ConvBertModel

[[autodoc]] ConvBertModel
    - forward

## ConvBertForMaskedLM

[[autodoc]] ConvBertForMaskedLM
    - forward

## ConvBertForSequenceClassification

[[autodoc]] ConvBertForSequenceClassification
    - forward

## ConvBertForMultipleChoice

[[autodoc]] ConvBertForMultipleChoice
    - forward

## ConvBertForTokenClassification

[[autodoc]] ConvBertForTokenClassification
    - forward

## ConvBertForQuestionAnswering

[[autodoc]] ConvBertForQuestionAnswering
    - forward

</pt>
<tf>

## TFConvBertModel

[[autodoc]] TFConvBertModel
    - call

## TFConvBertForMaskedLM

[[autodoc]] TFConvBertForMaskedLM
    - call

## TFConvBertForSequenceClassification

[[autodoc]] TFConvBertForSequenceClassification
    - call

## TFConvBertForMultipleChoice

[[autodoc]] TFConvBertForMultipleChoice
    - call

## TFConvBertForTokenClassification

[[autodoc]] TFConvBertForTokenClassification
    - call

## TFConvBertForQuestionAnswering

[[autodoc]] TFConvBertForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "def94f09-5597-45a5-85ef-6aaab359301b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e7d81cb44b28241fb97cecd8dd189527f3c1d841c6746057893dfc87956a47fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bd3f9099-a39e-4eb1-98be-cf48ffaaf83c", "node_type": "1", "metadata": {}, "hash": "5949f0fc9a6193534d047dde4720e0b6d9ec4a115216cc5a020a4b8f47b0b3d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3287, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SAM

## Overview

SAM (Segment Anything Model) was proposed in [Segment Anything](https://arxiv.org/pdf/2304.02643v1.pdf) by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick.

The model can be used to predict segmentation masks of any object of interest given an input image. 

![example image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/sam-output.png)

The abstract from the paper is the following:

*We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at [https://segment-anything.com](https://segment-anything.com) to foster research into foundation models for computer vision.*

Tips:

- The model predicts binary masks that states the presence or not of the object of interest given an image.
- The model predicts much better results if input 2D points and/or input bounding boxes are provided
- You can prompt multiple points for the same image, and predict a single mask. 
- Fine-tuning the model is not supported yet
- According to the paper, textual input should be also supported. However, at this time of writing this seems to be not supported according to [the official repository](https://github.com/facebookresearch/segment-anything/issues/4#issuecomment-1497626844). 


This model was contributed by [ybelkada](https://huggingface.co/ybelkada) and [ArthurZ](https://huggingface.co/ArthurZ).
The original code can be found [here](https://github.com/facebookresearch/segment-anything).

Below is an example on how to run mask generation given an image and a 2D point:

```python
import torch
from PIL import Image
import requests
from transformers import SamModel, SamProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"
model = SamModel.from_pretrained("facebook/sam-vit-huge").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
input_points = [[[450, 600]]]  # 2D location of a window in the image

inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)

masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu()
)
scores = outputs.iou_scores
```

You can also process your own masks alongside the input images in the processor to be passed to the model.


key: _node_content
string_value:
{"id_": "bd3f9099-a39e-4eb1-98be-cf48ffaaf83c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e7d81cb44b28241fb97cecd8dd189527f3c1d841c6746057893dfc87956a47fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "def94f09-5597-45a5-85ef-6aaab359301b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4240c60a84b9cdae77d1526bd7e05c29f24a85b4a06428ceb183d2717392cea1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "51a82e7f-497a-402a-b844-32e634bd52c1", "node_type": "1", "metadata": {}, "hash": "02f618440a7997e331ddc8f45c1cc1198aafa966f5f0a606ed8e6ebea7021633", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2014, "end_char_idx": 5003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [ybelkada](https://huggingface.co/ybelkada) and [ArthurZ](https://huggingface.co/ArthurZ).
The original code can be found [here](https://github.com/facebookresearch/segment-anything).

Below is an example on how to run mask generation given an image and a 2D point:

```python
import torch
from PIL import Image
import requests
from transformers import SamModel, SamProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"
model = SamModel.from_pretrained("facebook/sam-vit-huge").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
input_points = [[[450, 600]]]  # 2D location of a window in the image

inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)

masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu()
)
scores = outputs.iou_scores
```

You can also process your own masks alongside the input images in the processor to be passed to the model.

```python
import torch
from PIL import Image
import requests
from transformers import SamModel, SamProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"
model = SamModel.from_pretrained("facebook/sam-vit-huge").to(device)
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")
mask_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
segmentation_map = Image.open(requests.get(mask_url, stream=True).raw).convert("1")
input_points = [[[450, 600]]]  # 2D location of a window in the image

inputs = processor(raw_image, input_points=input_points, segmentation_maps=segmentation_map, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)

masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(), inputs["reshaped_input_sizes"].cpu()
)
scores = outputs.iou_scores
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SAM.

- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb) for using the model.
- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) for using the automatic mask generation pipeline.
- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb) for inference with MedSAM, a fine-tuned version of SAM on the medical domain.


key: _node_content
string_value:
{"id_": "51a82e7f-497a-402a-b844-32e634bd52c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e7d81cb44b28241fb97cecd8dd189527f3c1d841c6746057893dfc87956a47fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bd3f9099-a39e-4eb1-98be-cf48ffaaf83c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2e6d0638a44b0713f16ed7ee029283822fa0d928a56d099aacd30dccefbb77e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4491, "end_char_idx": 6712, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb) for using the model.
- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) for using the automatic mask generation pipeline.
- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb) for inference with MedSAM, a fine-tuned version of SAM on the medical domain. 🌎
- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb) for fine-tuning the model on custom data. 🌎

## SlimSAM

SlimSAM, a pruned version of SAM, was proposed in [0.1% Data Makes Segment Anything Slim](https://arxiv.org/abs/2312.05284) by Zigeng Chen et al. SlimSAM reduces the size of the SAM models considerably while maintaining the same performance.

Checkpoints can be found on the [hub](https://huggingface.co/models?other=slimsam), and they can be used as a drop-in replacement of SAM.

## Grounded SAM

One can combine [Grounding DINO](grounding-dino) with SAM for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) 🌍 for details.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png"
alt="drawing" width="900"/>

<small> Grounded SAM overview. Taken from the <a href="https://github.com/IDEA-Research/Grounded-Segment-Anything">original repository</a>. </small>

## SamConfig

[[autodoc]] SamConfig

## SamVisionConfig

[[autodoc]] SamVisionConfig

## SamMaskDecoderConfig

[[autodoc]] SamMaskDecoderConfig

## SamPromptEncoderConfig

[[autodoc]] SamPromptEncoderConfig


## SamProcessor

[[autodoc]] SamProcessor


## SamImageProcessor

[[autodoc]] SamImageProcessor


## SamModel

[[autodoc]] SamModel
    - forward


## TFSamModel

[[autodoc]] TFSamModel
    - call


key: _node_content
string_value:
{"id_": "b7ee07d4-f8d1-488b-a940-9af2bb0ca127", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon/", "title": "Falcon", "tokens": 708, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "94b002ce-2c4f-45d6-b699-88faa6d2b303", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon/", "title": "Falcon", "tokens": 708, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "46b417e980e249b2c735af5b0b39c6ef427a9374b53ec91433f4e8ef45b71464", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2444, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Falcon

## Overview

Falcon is a class of causal decoder-only models built by [TII](https://www.tii.ae/). The largest Falcon checkpoints
have been trained on >=1T tokens of text, with a particular emphasis on the [RefinedWeb](https://arxiv.org/abs/2306.01116)
corpus. They are made available under the Apache 2.0 license.


Falcon's architecture is modern and optimized for inference, with multi-query attention and support for efficient
attention variants like `FlashAttention`. Both 'base' models trained only as causal language models as well as
'instruct' models that have received further fine-tuning are available.


Falcon models are (as of 2023) some of the largest and most powerful open-source language models,
and consistently rank highly in the [OpenLLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).

## Converting custom checkpoints 

<Tip>

Falcon models were initially added to the Hugging Face Hub as custom code checkpoints. However, Falcon is now fully
supported in the Transformers library. If you fine-tuned a model from a custom code checkpoint, we recommend converting
your checkpoint to the new in-library format, as this should give significant improvements to stability and
performance, especially for generation, as well as removing the need to use `trust_remote_code=True`!

</Tip>

You can convert custom code checkpoints to full Transformers checkpoints using the `convert_custom_code_checkpoint.py` 
script located in the
[Falcon model directory](https://github.com/huggingface/transformers/tree/main/src/transformers/models/falcon)
of the Transformers library. To use this script, simply call it with 
`python convert_custom_code_checkpoint.py --checkpoint_dir my_model`. This will convert your checkpoint in-place, and
you can immediately load it from the directory afterwards with e.g. `from_pretrained()`. If your model hasn't been
uploaded to the Hub, we recommend making a backup before attempting the conversion, just in case!


## FalconConfig

[[autodoc]] FalconConfig
    - all

## FalconModel

[[autodoc]] FalconModel
    - forward

## FalconForCausalLM

[[autodoc]] FalconForCausalLM
    - forward

## FalconForSequenceClassification

[[autodoc]] FalconForSequenceClassification
    - forward

## FalconForTokenClassification

[[autodoc]] FalconForTokenClassification
    - forward

## FalconForQuestionAnswering

[[autodoc]] FalconForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "83fa7b51-dda6-4992-be6c-cf7d324bd4cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/barthez/", "title": "BARThez", "tokens": 728, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "812fe208-0d04-41f1-95dc-4e6c34559978", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/barthez/", "title": "BARThez", "tokens": 728, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bca1383888182f628b8ad00e6f0a808ed6b18611ed9ca8461ee8f2bd120cc488", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BARThez

## Overview

The BARThez model was proposed in [BARThez: a Skilled Pretrained French Sequence-to-Sequence Model](https://arxiv.org/abs/2010.12321) by Moussa Kamal Eddine, Antoine J.-P. Tixier, Michalis Vazirgiannis on 23 Oct,
2020.

The abstract of the paper:


*Inductive transfer learning, enabled by self-supervised learning, have taken the entire Natural Language Processing
(NLP) field by storm, with models such as BERT and BART setting new state of the art on countless natural language
understanding tasks. While there are some notable exceptions, most of the available models and research have been
conducted for the English language. In this work, we introduce BARThez, the first BART model for the French language
(to the best of our knowledge). BARThez was pretrained on a very large monolingual French corpus from past research
that we adapted to suit BART's perturbation schemes. Unlike already existing BERT-based French language models such as
CamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks, since not only its encoder but also
its decoder is pretrained. In addition to discriminative tasks from the FLUE benchmark, we evaluate BARThez on a novel
summarization dataset, OrangeSum, that we release with this paper. We also continue the pretraining of an already
pretrained multilingual BART on BARThez's corpus, and we show that the resulting model, which we call mBARTHez,
provides a significant boost over vanilla BARThez, and is on par with or outperforms CamemBERT and FlauBERT.*

This model was contributed by [moussakam](https://huggingface.co/moussakam). The Authors' code can be found [here](https://github.com/moussaKam/BARThez).

<Tip> 

BARThez implementation is the same as BART, except for tokenization. Refer to [BART documentation](bart) for information on 
configuration classes and their parameters. BARThez-specific tokenizers are documented below.  

</Tip>

## Resources

- BARThez can be fine-tuned on sequence-to-sequence tasks in a similar way as BART, check:
  [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).


## BarthezTokenizer

[[autodoc]] BarthezTokenizer

## BarthezTokenizerFast

[[autodoc]] BarthezTokenizerFast


key: _node_content
string_value:
{"id_": "1e7068fe-95c6-43b6-9e12-a07250e506a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "12748c3341bcd02d65289074a4d482070f3abc0ed642718db9f9c7aecb7157c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "395d2acd-0251-4e03-9db7-a4e25b0b1beb", "node_type": "1", "metadata": {}, "hash": "71230b91dfc3590aa82329aa8db5865bdc3b280da566a97ddf023e647cce60ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3455, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LUKE

## Overview

The LUKE model was proposed in [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://arxiv.org/abs/2010.01057) by Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda and Yuji Matsumoto.
It is based on RoBERTa and adds entity embeddings as well as an entity-aware self-attention mechanism, which helps
improve performance on various downstream tasks involving reasoning about entities such as named entity recognition,
extractive and cloze-style question answering, entity typing, and relation classification.

The abstract from the paper is the following:

*Entity representations are useful in natural language tasks involving entities. In this paper, we propose new
pretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed
model treats words and entities in a given text as independent tokens, and outputs contextualized representations of
them. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves
predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also
propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the
transformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model
achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains
state-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification),
CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question
answering).*

This model was contributed by [ikuyamada](https://huggingface.co/ikuyamada) and [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/studio-ousia/luke).

## Usage tips

- This implementation is the same as [`RobertaModel`] with the addition of entity embeddings as well
  as an entity-aware self-attention mechanism, which improves performance on tasks involving reasoning about entities.
- LUKE treats entities as input tokens; therefore, it takes `entity_ids`, `entity_attention_mask`,
  `entity_token_type_ids` and `entity_position_ids` as extra input. You can obtain those using
  [`LukeTokenizer`].
- [`LukeTokenizer`] takes `entities` and `entity_spans` (character-based start and end
  positions of the entities in the input text) as extra input. `entities` typically consist of [MASK] entities or
  Wikipedia entities. The brief description when inputting these entities are as follows:

  - *Inputting [MASK] entities to compute entity representations*: The [MASK] entity is used to mask entities to be
    predicted during pretraining. When LUKE receives the [MASK] entity, it tries to predict the original entity by
    gathering the information about the entity from the input text. Therefore, the [MASK] entity can be used to address
    downstream tasks requiring the information of entities in text such as entity typing, relation classification, and
    named entity recognition.
  - *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information
    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding.


key: _node_content
string_value:
{"id_": "395d2acd-0251-4e03-9db7-a4e25b0b1beb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "12748c3341bcd02d65289074a4d482070f3abc0ed642718db9f9c7aecb7157c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1e7068fe-95c6-43b6-9e12-a07250e506a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6601764a885ca8f7100eae75886c48c9d4d81b89bf33a528db84237ee64e9ab9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7d66a52-78c9-4734-ad5a-c43745556fec", "node_type": "1", "metadata": {}, "hash": "5eefc557c728e86ebab5a622ad313d9972541c66d04c8d7d929cac4455b09d29", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1780, "end_char_idx": 5005, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

This model was contributed by [ikuyamada](https://huggingface.co/ikuyamada) and [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/studio-ousia/luke).

## Usage tips

- This implementation is the same as [`RobertaModel`] with the addition of entity embeddings as well
  as an entity-aware self-attention mechanism, which improves performance on tasks involving reasoning about entities.
- LUKE treats entities as input tokens; therefore, it takes `entity_ids`, `entity_attention_mask`,
  `entity_token_type_ids` and `entity_position_ids` as extra input. You can obtain those using
  [`LukeTokenizer`].
- [`LukeTokenizer`] takes `entities` and `entity_spans` (character-based start and end
  positions of the entities in the input text) as extra input. `entities` typically consist of [MASK] entities or
  Wikipedia entities. The brief description when inputting these entities are as follows:

  - *Inputting [MASK] entities to compute entity representations*: The [MASK] entity is used to mask entities to be
    predicted during pretraining. When LUKE receives the [MASK] entity, it tries to predict the original entity by
    gathering the information about the entity from the input text. Therefore, the [MASK] entity can be used to address
    downstream tasks requiring the information of entities in text such as entity typing, relation classification, and
    named entity recognition.
  - *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information
    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding. By
    using Wikipedia entities as input tokens, LUKE outputs token representations enriched by the information stored in
    the embeddings of these entities. This is particularly effective for tasks requiring real-world knowledge, such as
    question answering.

- There are three head models for the former use case:

  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as
    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).
    This model places a linear head on top of the output entity representation.
  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities
    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This
    model places a linear head on top of the concatenated output representation of the pair of given entities.
  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as
    named entity recognition (NER). This model places a linear head on top of the output entity representations. You
    can address NER using this model by inputting all possible entity spans in the text to the model.

  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these
  head models by specifying `task="entity_classification"`, `task="entity_pair_classification"`, or
  `task="entity_span_classification"`. Please refer to the example code of each head models.


key: _node_content
string_value:
{"id_": "c7d66a52-78c9-4734-ad5a-c43745556fec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "12748c3341bcd02d65289074a4d482070f3abc0ed642718db9f9c7aecb7157c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "395d2acd-0251-4e03-9db7-a4e25b0b1beb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ed9dcf1668e4b0e353a794976564949ce7262663a45202db72210551a11b5847", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6435e2c8-b753-4d6f-aaab-3700c06ab081", "node_type": "1", "metadata": {}, "hash": "8c707e36fe102771bbe4e5cc0da4f9dd23ad91bc0fb9d9e7a8fe31bdb21c8be1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3226, "end_char_idx": 5410, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information
    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding. By
    using Wikipedia entities as input tokens, LUKE outputs token representations enriched by the information stored in
    the embeddings of these entities. This is particularly effective for tasks requiring real-world knowledge, such as
    question answering.

- There are three head models for the former use case:

  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as
    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).
    This model places a linear head on top of the output entity representation.
  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities
    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This
    model places a linear head on top of the concatenated output representation of the pair of given entities.
  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as
    named entity recognition (NER). This model places a linear head on top of the output entity representations. You
    can address NER using this model by inputting all possible entity spans in the text to the model.

  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these
  head models by specifying `task="entity_classification"`, `task="entity_pair_classification"`, or
  `task="entity_span_classification"`. Please refer to the example code of each head models.

Usage example:

```python
>>> from transformers import LukeTokenizer, LukeModel, LukeForEntityPairClassification

>>> model = LukeModel.from_pretrained("studio-ousia/luke-base")
>>> tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-base")
# Example 1: Computing the contextualized entity representation corresponding to the entity mention "Beyoncé"

>>> text = "Beyoncé lives in Los Angeles."


key: _node_content
string_value:
{"id_": "6435e2c8-b753-4d6f-aaab-3700c06ab081", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "12748c3341bcd02d65289074a4d482070f3abc0ed642718db9f9c7aecb7157c6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c7d66a52-78c9-4734-ad5a-c43745556fec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "045668b281256d3d100d77448ee7b23dafbe0fe9d76b7df22a0f91cadc8f0cca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3616, "end_char_idx": 8615, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is particularly effective for tasks requiring real-world knowledge, such as
    question answering.

- There are three head models for the former use case:

  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as
    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).
    This model places a linear head on top of the output entity representation.
  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities
    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This
    model places a linear head on top of the concatenated output representation of the pair of given entities.
  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as
    named entity recognition (NER). This model places a linear head on top of the output entity representations. You
    can address NER using this model by inputting all possible entity spans in the text to the model.

  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these
  head models by specifying `task="entity_classification"`, `task="entity_pair_classification"`, or
  `task="entity_span_classification"`. Please refer to the example code of each head models.

Usage example:

```python
>>> from transformers import LukeTokenizer, LukeModel, LukeForEntityPairClassification

>>> model = LukeModel.from_pretrained("studio-ousia/luke-base")
>>> tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-base")
# Example 1: Computing the contextualized entity representation corresponding to the entity mention "Beyoncé"

>>> text = "Beyoncé lives in Los Angeles."
>>> entity_spans = [(0, 7)]  # character-based entity span corresponding to "Beyoncé"
>>> inputs = tokenizer(text, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> word_last_hidden_state = outputs.last_hidden_state
>>> entity_last_hidden_state = outputs.entity_last_hidden_state
# Example 2: Inputting Wikipedia entities to obtain enriched contextualized representations

>>> entities = [
...     "Beyoncé",
...     "Los Angeles",
... ]  # Wikipedia entity titles corresponding to the entity mentions "Beyoncé" and "Los Angeles"
>>> entity_spans = [(0, 7), (17, 28)]  # character-based entity spans corresponding to "Beyoncé" and "Los Angeles"
>>> inputs = tokenizer(text, entities=entities, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> word_last_hidden_state = outputs.last_hidden_state
>>> entity_last_hidden_state = outputs.entity_last_hidden_state
# Example 3: Classifying the relationship between two entities using LukeForEntityPairClassification head model

>>> model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
>>> tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
>>> entity_spans = [(0, 7), (17, 28)]  # character-based entity spans corresponding to "Beyoncé" and "Los Angeles"
>>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt")
>>> outputs = model(**inputs)
>>> logits = outputs.logits
>>> predicted_class_idx = int(logits[0].argmax())
>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
```

## Resources

- [A demo notebook on how to fine-tune [`LukeForEntityPairClassification`] for relation classification](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LUKE)
- [Notebooks showcasing how you to reproduce the results as reported in the paper with the HuggingFace implementation of LUKE](https://github.com/studio-ousia/luke/tree/master/notebooks)
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## LukeConfig

[[autodoc]] LukeConfig

## LukeTokenizer

[[autodoc]] LukeTokenizer
    - __call__
    - save_vocabulary

## LukeModel

[[autodoc]] LukeModel
    - forward

## LukeForMaskedLM

[[autodoc]] LukeForMaskedLM
    - forward

## LukeForEntityClassification

[[autodoc]] LukeForEntityClassification
    - forward

## LukeForEntityPairClassification

[[autodoc]] LukeForEntityPairClassification
    - forward

## LukeForEntitySpanClassification

[[autodoc]] LukeForEntitySpanClassification
    - forward

## LukeForSequenceClassification

[[autodoc]] LukeForSequenceClassification
    - forward

## LukeForMultipleChoice

[[autodoc]] LukeForMultipleChoice
    - forward

## LukeForTokenClassification

[[autodoc]] LukeForTokenClassification
    - forward

## LukeForQuestionAnswering

[[autodoc]] LukeForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "d35c4065-0f08-476d-8404-62257d86252b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/focalnet/", "title": "FocalNet", "tokens": 878, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a1b7d5ff-909c-4bc8-88eb-321fd0edc15c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/focalnet/", "title": "FocalNet", "tokens": 878, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3e1b2be1de8c97476b45eb8335b49f324367e8e585249a548dbb17c107f3b6a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FocalNet

## Overview

The FocalNet model was proposed in [Focal Modulation Networks](https://arxiv.org/abs/2203.11926) by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao.
FocalNets completely replace self-attention (used in models like [ViT](vit) and [Swin](swin)) by a focal modulation mechanism for modeling token interactions in vision.
The authors claim that FocalNets outperform self-attention based models with similar computational costs on the tasks of image classification, object detection, and segmentation.

The abstract from the paper is the following:

*We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation mechanism for modeling token interactions in vision. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (ii) gated aggregation to selectively gather contexts for each query token based on its
content, and (iii) element-wise modulation or affine transformation to inject the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational costs on the tasks of image classification, object detection, and segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K in 224 resolution, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 and 384, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1\times outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3\times schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and Mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. Using huge FocalNet and DINO, we achieved 64.3 and 64.4 mAP on COCO minival and test-dev, respectively, establishing new SoTA on top of much larger attention-based models like Swinv2-G and BEIT-3.*

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/FocalNet).

## FocalNetConfig

[[autodoc]] FocalNetConfig

## FocalNetModel

[[autodoc]] FocalNetModel
    - forward

## FocalNetForMaskedImageModeling

[[autodoc]] FocalNetForMaskedImageModeling
    - forward

## FocalNetForImageClassification

[[autodoc]] FocalNetForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "0131c2ad-a0e3-4af7-a068-52f8d8738ddb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie/", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7d456e14-5b42-44d5-808c-0143fd69bcbc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie/", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e3cc8a50b5f2dd1f4db70c6ad4644f803151e33717dc0dad9e55bd9122c4e9dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "08430799-837e-4201-86de-dab6aa608023", "node_type": "1", "metadata": {}, "hash": "623aa9ce8663614044bb6a1d57fd891e2a89fa4a98813feb1753d1e2a7f76554", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2016, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ERNIE

## Overview
ERNIE is a series of powerful models proposed by baidu, especially in Chinese tasks,
including [ERNIE1.0](https://arxiv.org/abs/1904.09223), [ERNIE2.0](https://ojs.aaai.org/index.php/AAAI/article/view/6428),
[ERNIE3.0](https://arxiv.org/abs/2107.02137), [ERNIE-Gram](https://arxiv.org/abs/2010.12148), [ERNIE-health](https://arxiv.org/abs/2110.07244), etc.

These models are contributed by [nghuyong](https://huggingface.co/nghuyong) and the official code can be found in [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) (in PaddlePaddle).

### Usage example
Take `ernie-1.0-base-zh` as an example:

```Python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
model = AutoModel.from_pretrained("nghuyong/ernie-1.0-base-zh")
```

### Model checkpoints

|     Model Name      | Language |           Description           |
|:-------------------:|:--------:|:-------------------------------:|
|  ernie-1.0-base-zh  | Chinese  | Layer:12, Heads:12, Hidden:768  |
|  ernie-2.0-base-en  | English  | Layer:12, Heads:12, Hidden:768  |
| ernie-2.0-large-en  | English  | Layer:24, Heads:16, Hidden:1024 |
|  ernie-3.0-base-zh  | Chinese  | Layer:12, Heads:12, Hidden:768  |
| ernie-3.0-medium-zh | Chinese  |  Layer:6, Heads:12, Hidden:768  |
|  ernie-3.0-mini-zh  | Chinese  |  Layer:6, Heads:12, Hidden:384  |
| ernie-3.0-micro-zh  | Chinese  |  Layer:4, Heads:12, Hidden:384  |
|  ernie-3.0-nano-zh  | Chinese  |  Layer:4, Heads:12, Hidden:312  |
|   ernie-health-zh   | Chinese  | Layer:12, Heads:12, Hidden:768  |
|    ernie-gram-zh    | Chinese  | Layer:12, Heads:12, Hidden:768  |

You can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official
repo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html)
and [ERNIE](https://github.com/PaddlePaddle/ERNIE/blob/repro).


key: _node_content
string_value:
{"id_": "08430799-837e-4201-86de-dab6aa608023", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie/", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7d456e14-5b42-44d5-808c-0143fd69bcbc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie/", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e3cc8a50b5f2dd1f4db70c6ad4644f803151e33717dc0dad9e55bd9122c4e9dc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0131c2ad-a0e3-4af7-a068-52f8d8738ddb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie/", "title": "ERNIE", "tokens": 1181, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c3d179efce3b2409f43a73cfc0afd32bdc3597e5a63c6d5911905d6561981ef4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2018, "end_char_idx": 3262, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## ErnieConfig

[[autodoc]] ErnieConfig
    - all

## Ernie specific outputs

[[autodoc]] models.ernie.modeling_ernie.ErnieForPreTrainingOutput

## ErnieModel

[[autodoc]] ErnieModel
    - forward

## ErnieForPreTraining

[[autodoc]] ErnieForPreTraining
    - forward

## ErnieForCausalLM

[[autodoc]] ErnieForCausalLM
    - forward

## ErnieForMaskedLM

[[autodoc]] ErnieForMaskedLM
    - forward

## ErnieForNextSentencePrediction

[[autodoc]] ErnieForNextSentencePrediction
    - forward

## ErnieForSequenceClassification

[[autodoc]] ErnieForSequenceClassification
    - forward

## ErnieForMultipleChoice

[[autodoc]] ErnieForMultipleChoice
    - forward

## ErnieForTokenClassification

[[autodoc]] ErnieForTokenClassification
    - forward

## ErnieForQuestionAnswering

[[autodoc]] ErnieForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "0e130dad-4fc4-4503-a6b2-d87b51d02fec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "585d993d-1ad2-4fa5-aec1-6c2a32b875c0", "node_type": "1", "metadata": {}, "hash": "d3318ccb681b9d82e824d6673cd36977ebfc8fc580b91791bef9370aa4a7e2a6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3522, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLIP

## Overview

The CLIP model was proposed in [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020) by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. CLIP
(Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be
instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing
for the task, similarly to the zero-shot capabilities of GPT-2 and 3.

The abstract from the paper is the following:

*State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This
restricted form of supervision limits their generality and usability since additional labeled data is needed to specify
any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a
much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes
with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400
million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference
learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study
the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks
such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The
model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need
for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot
without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained
model weights at this https URL.*

This model was contributed by [valhalla](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/openai/CLIP).

## Usage tips and example

CLIP is a multi-modal vision and language model. It can be used for image-text similarity and for zero-shot image
classification. CLIP uses a ViT like transformer to get visual features and a causal language model to get the text
features. Both the text and visual features are then projected to a latent space with identical dimension. The dot
product between the projected image and text features is then used as a similar score.

To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,
which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image. The authors
also add absolute position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder.
The [`CLIPImageProcessor`] can be used to resize (or rescale) and normalize images for the model.

The [`CLIPTokenizer`] is used to encode the text. The [`CLIPProcessor`] wraps
[`CLIPImageProcessor`] and [`CLIPTokenizer`] into a single instance to both
encode the text and prepare the images. The following example shows how to get the image-text similarity scores using
[`CLIPProcessor`] and [`CLIPModel`].


key: _node_content
string_value:
{"id_": "585d993d-1ad2-4fa5-aec1-6c2a32b875c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e130dad-4fc4-4503-a6b2-d87b51d02fec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5e9820f14cec74796ab72aba16b407658ff96cb0819071db9b015de0bb62a94b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "929d10c0-fdc9-4a42-ba1b-9c8350f77b74", "node_type": "1", "metadata": {}, "hash": "8c9b609ffaa33c00e18c148884ad24c43adb5f9aa490f301120f904ce5aceecf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3525, "end_char_idx": 6715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> from PIL import Image
>>> import requests

>>> from transformers import CLIPProcessor, CLIPModel

>>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
>>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
```


### Combining CLIP and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16`)

<Tip warning={true}>

For small batch sizes, you might notice a slowdown in your model when using flash attention. Refer to the section [Expected speedups with Flash Attention and SDPA](#Expected-speedups-with-Flash-Attention-and-SDPA) below and select an appropriate attention implementation.

</Tip>

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> import requests
>>> from PIL import Image

>>> from transformers import CLIPProcessor, CLIPModel

>>> device = "cuda"
>>> torch_dtype = torch.float16

>>> model = CLIPModel.from_pretrained(
...     "openai/clip-vit-base-patch32",
...     attn_implementation="flash_attention_2",
...     device_map=device,
...     torch_dtype=torch_dtype,
... )
>>> processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
>>> inputs.to(device)

>>> with torch.no_grad():
...     with torch.autocast(device):
...         outputs = model(**inputs)

>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities
>>> print(probs)
tensor([[0.9946, 0.0052]], device='cuda:0', dtype=torch.float16)
```


### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.


key: _node_content
string_value:
{"id_": "929d10c0-fdc9-4a42-ba1b-9c8350f77b74", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "585d993d-1ad2-4fa5-aec1-6c2a32b875c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5610e18a35ebbe6a60e3a70d792ba707c20b1dba0f0b0ddd6ceebbfd63710297", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7fd9fec-8b4f-43b1-b1aa-980b56081f6c", "node_type": "1", "metadata": {}, "hash": "52da01a60c34814bc5363d1e8e3352c2f893e8e81bcac1d0ac62ec43a933a6f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6156, "end_char_idx": 8282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```python
from transformers import CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32", torch_dtype=torch.float16, attn_implementation="sdpa")
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

### Expected speedups with Flash Attention and SDPA

On a local benchmark (NVIDIA A10G, PyTorch 2.3.1+cu121) with `float16`, we saw the following speedups during inference for `"openai/clip-vit-large-patch14"` checkpoint ([code](https://gist.github.com/qubvel/ac691a54e54f9fae8144275f866a7ff8)):

#### CLIPTextModel

|   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |
|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|
|                 4 |            0.009 |          0.012 |         0.737 |           0.007 |          1.269 |
|                16 |            0.009 |          0.014 |         0.659 |           0.008 |          1.187 |
|                32 |            0.018 |          0.021 |         0.862 |           0.016 |          1.142 |
|                64 |            0.034 |          0.034 |         1.001 |           0.03  |          1.163 |
|               128 |            0.063 |          0.058 |         1.09  |           0.054 |          1.174 |

!


key: _node_content
string_value:
{"id_": "f7fd9fec-8b4f-43b1-b1aa-980b56081f6c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "929d10c0-fdc9-4a42-ba1b-9c8350f77b74", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "febc2eee619953913ee4b73fcd1603d3bd82439276eb65b7366533c2d1a7b27a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "681581cb-bdca-49ba-a625-4e65502c0737", "node_type": "1", "metadata": {}, "hash": "3e258336f1bf3c9df50ab201ba1d8fa3c945905a818c294594b7f92f29f4d26b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7161, "end_char_idx": 9072, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`torch.float16` or `torch.bfloat16`).

### Expected speedups with Flash Attention and SDPA

On a local benchmark (NVIDIA A10G, PyTorch 2.3.1+cu121) with `float16`, we saw the following speedups during inference for `"openai/clip-vit-large-patch14"` checkpoint ([code](https://gist.github.com/qubvel/ac691a54e54f9fae8144275f866a7ff8)):

#### CLIPTextModel

|   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |
|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|
|                 4 |            0.009 |          0.012 |         0.737 |           0.007 |          1.269 |
|                16 |            0.009 |          0.014 |         0.659 |           0.008 |          1.187 |
|                32 |            0.018 |          0.021 |         0.862 |           0.016 |          1.142 |
|                64 |            0.034 |          0.034 |         1.001 |           0.03  |          1.163 |
|               128 |            0.063 |          0.058 |         1.09  |           0.054 |          1.174 |

![clip_text_model_viz_3](https://github.com/user-attachments/assets/e9826b43-4e66-4f4c-952b-af4d90bd38eb)

#### CLIPVisionModel

|   Image batch size |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |
|-------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|
|                  1 |            0.016 |          0.013 |         1.247 |           0.012 |          1.318 |
|                  4 |            0.025 |          0.021 |         1.198 |           0.021 |          1.202 |
|                 16 |            0.093 |          0.075 |         1.234 |           0.075 |          1.24  |
|                 32 |            0.181 |          0.147 |         1.237 |           0.146 |          1.241 |

!


key: _node_content
string_value:
{"id_": "681581cb-bdca-49ba-a625-4e65502c0737", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7fd9fec-8b4f-43b1-b1aa-980b56081f6c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "625b52740faa03f5cdf2c9fcc696341e2abdcae7567562a06bf783da8a78d6d2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "65087b3b-39f4-4856-b069-6d8a438d7e02", "node_type": "1", "metadata": {}, "hash": "eb72e2bfcb67fdec0622a02fda7064d4fcff0228fc7028797017f946b95a1a04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8282, "end_char_idx": 11137, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[clip_text_model_viz_3](https://github.com/user-attachments/assets/e9826b43-4e66-4f4c-952b-af4d90bd38eb)

#### CLIPVisionModel

|   Image batch size |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |
|-------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|
|                  1 |            0.016 |          0.013 |         1.247 |           0.012 |          1.318 |
|                  4 |            0.025 |          0.021 |         1.198 |           0.021 |          1.202 |
|                 16 |            0.093 |          0.075 |         1.234 |           0.075 |          1.24  |
|                 32 |            0.181 |          0.147 |         1.237 |           0.146 |          1.241 |

![clip_image_model_viz_3](https://github.com/user-attachments/assets/50a36206-e3b9-4adc-ac8e-926b8b071d63)

#### CLIPModel

|   Image batch size |   Num text labels |   Eager (s/iter) |   FA2 (s/iter) |   FA2 speedup |   SDPA (s/iter) |   SDPA speedup |
|-------------------:|------------------:|-----------------:|---------------:|--------------:|----------------:|---------------:|
|                  1 |                 4 |            0.025 |          0.026 |         0.954 |           0.02  |          1.217 |
|                  1 |                16 |            0.026 |          0.028 |         0.918 |           0.02  |          1.287 |
|                  1 |                64 |            0.042 |          0.046 |         0.906 |           0.036 |          1.167 |
|                  4 |                 4 |            0.028 |          0.033 |         0.849 |           0.024 |          1.189 |
|                  4 |                16 |            0.034 |          0.035 |         0.955 |           0.029 |          1.169 |
|                  4 |                64 |            0.059 |          0.055 |         1.072 |           0.05  |          1.179 |
|                 16 |                 4 |            0.096 |          0.088 |         1.091 |           0.078 |          1.234 |
|                 16 |                16 |            0.102 |          0.09  |         1.129 |           0.083 |          1.224 |
|                 16 |                64 |            0.127 |          0.11  |         1.157 |           0.105 |          1.218 |
|                 32 |                 4 |            0.185 |          0.159 |         1.157 |           0.149 |          1.238 |
|                 32 |                16 |            0.19  |          0.162 |         1.177 |           0.154 |          1.233 |
|                 32 |                64 |            0.216 |          0.181 |         1.19  |           0.176 |          1.228 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with CLIP.


key: _node_content
string_value:
{"id_": "65087b3b-39f4-4856-b069-6d8a438d7e02", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "681581cb-bdca-49ba-a625-4e65502c0737", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fa1170df597a6c239edb6f7bd033bc0045296bf950dd71f1c69c9efc4e0ede3c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e390a74c-9232-46d9-89ca-9f7a0a073f03", "node_type": "1", "metadata": {}, "hash": "596220c249636460c7095edfb37084f21d7f4482dfe4c66836377ac426b2cec0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11139, "end_char_idx": 13263, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Fine tuning CLIP with Remote Sensing (Satellite) images and captions](https://huggingface.co/blog/fine-tune-clip-rsicd), a blog post about how to fine-tune CLIP with [RSICD dataset](https://github.com/201528014227051/RSICD_optimal) and comparison of performance changes due to data augmentation.
- This [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text) shows how to train a CLIP-like vision-text dual encoder model using a pre-trained vision and text encoder using [COCO dataset](https://cocodataset.org/#home).

<PipelineTag pipeline="image-to-text"/>

- A [notebook](https://colab.research.google.com/drive/1tuoAC5F4sC7qid56Z0ap-stR3rwdk0ZV?usp=sharing) on how to use a pretrained CLIP for inference with beam search for image captioning. 🌎

**Image retrieval**

- A [notebook](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing) on image retrieval using pretrained CLIP and computing MRR(Mean Reciprocal Rank) score. 🌎
- A [notebook](https://colab.research.google.com/github/deep-diver/image_search_with_natural_language/blob/main/notebooks/Image_Search_CLIP.ipynb) on image retrieval and showing the similarity score. 🌎
- A [notebook](https://colab.research.google.com/drive/1xO-wC_m_GNzgjIBQ4a4znvQkvDoZJvH4?usp=sharing) on how to map images and texts to the same vector space using Multilingual CLIP. 🌎 
- A [notebook](https://colab.research.google.com/github/vivien000/clip-demo/blob/master/clip.ipynb#scrollTo=uzdFhRGqiWkR) on how to run CLIP on semantic image search using [Unsplash](https://unsplash.com) and [TMDB](https://www.themoviedb.org/) datasets. 🌎

**Explainability**

- A [notebook](https://colab.research.google.com/github/hila-chefer/Transformer-MM-Explainability/blob/main/CLIP_explainability.ipynb) on how to visualize similarity between input token and image segment. 🌎

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "e390a74c-9232-46d9-89ca-9f7a0a073f03", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "445ab52b-776a-4489-ab49-4ec14c130bc7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "690cbff965f003883268b13f23bd218fa31c9a2632e817ea60e995c7155fed6e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65087b3b-39f4-4856-b069-6d8a438d7e02", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clip/", "title": "CLIP", "tokens": 4185, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "801799fba8374361410cf19db85b9e9ed501419aff3bab0204903c618b04a554", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11945, "end_char_idx": 14972, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🌎

**Image retrieval**

- A [notebook](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing) on image retrieval using pretrained CLIP and computing MRR(Mean Reciprocal Rank) score. 🌎
- A [notebook](https://colab.research.google.com/github/deep-diver/image_search_with_natural_language/blob/main/notebooks/Image_Search_CLIP.ipynb) on image retrieval and showing the similarity score. 🌎
- A [notebook](https://colab.research.google.com/drive/1xO-wC_m_GNzgjIBQ4a4znvQkvDoZJvH4?usp=sharing) on how to map images and texts to the same vector space using Multilingual CLIP. 🌎 
- A [notebook](https://colab.research.google.com/github/vivien000/clip-demo/blob/master/clip.ipynb#scrollTo=uzdFhRGqiWkR) on how to run CLIP on semantic image search using [Unsplash](https://unsplash.com) and [TMDB](https://www.themoviedb.org/) datasets. 🌎

**Explainability**

- A [notebook](https://colab.research.google.com/github/hila-chefer/Transformer-MM-Explainability/blob/main/CLIP_explainability.ipynb) on how to visualize similarity between input token and image segment. 🌎

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.

## CLIPConfig

[[autodoc]] CLIPConfig
    - from_text_vision_configs

## CLIPTextConfig

[[autodoc]] CLIPTextConfig

## CLIPVisionConfig

[[autodoc]] CLIPVisionConfig

## CLIPTokenizer

[[autodoc]] CLIPTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## CLIPTokenizerFast

[[autodoc]] CLIPTokenizerFast

## CLIPImageProcessor

[[autodoc]] CLIPImageProcessor
    - preprocess

## CLIPFeatureExtractor

[[autodoc]] CLIPFeatureExtractor

## CLIPProcessor

[[autodoc]] CLIPProcessor

<frameworkcontent>
<pt>

## CLIPModel

[[autodoc]] CLIPModel
    - forward
    - get_text_features
    - get_image_features

## CLIPTextModel

[[autodoc]] CLIPTextModel
    - forward

## CLIPTextModelWithProjection

[[autodoc]] CLIPTextModelWithProjection
    - forward

## CLIPVisionModelWithProjection

[[autodoc]] CLIPVisionModelWithProjection
    - forward

## CLIPVisionModel

[[autodoc]] CLIPVisionModel
    - forward

## CLIPForImageClassification

[[autodoc]] CLIPForImageClassification
    - forward

</pt>
<tf>

## TFCLIPModel

[[autodoc]] TFCLIPModel
    - call
    - get_text_features
    - get_image_features

## TFCLIPTextModel

[[autodoc]] TFCLIPTextModel
    - call

## TFCLIPVisionModel

[[autodoc]] TFCLIPVisionModel
    - call

</tf>
<jax>

## FlaxCLIPModel

[[autodoc]] FlaxCLIPModel
    - __call__
    - get_text_features
    - get_image_features

## FlaxCLIPTextModel

[[autodoc]] FlaxCLIPTextModel
    - __call__

## FlaxCLIPTextModelWithProjection

[[autodoc]] FlaxCLIPTextModelWithProjection
    - __call__

## FlaxCLIPVisionModel

[[autodoc]] FlaxCLIPVisionModel
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e9d545c8-1d0e-4188-af29-a4f964320f2e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14a0d8a7-c7fa-4d72-a4b1-005358c62fdb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19aa0f1993bb6090ee0e4a55d49e3671c5f2817609464a59dc3b7e8a89ca68ea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5362b806-a334-4c05-8c9a-803ba88c0771", "node_type": "1", "metadata": {}, "hash": "20a1b526ff550aca1c48e43c3a93e72c0b853ce8b92f07de6a3a3e1f2bad3486", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RT-DETR

## Overview


The RT-DETR model was proposed in [DETRs Beat YOLOs on Real-time Object Detection](https://arxiv.org/abs/2304.08069) by Wenyu Lv, Yian Zhao, Shangliang Xu, Jinman Wei, Guanzhong Wang, Cheng Cui, Yuning Du, Qingqing Dang, Yi Liu.

RT-DETR is an object detection model that stands for "Real-Time DEtection Transformer." This model is designed to perform object detection tasks with a focus on achieving real-time performance while maintaining high accuracy. Leveraging the transformer architecture, which has gained significant popularity in various fields of deep learning, RT-DETR processes images to identify and locate multiple objects within them.

The abstract from the paper is the following:

*Recently, end-to-end transformer-based detectors (DETRs) have achieved remarkable performance. However, the issue of the high computational cost of DETRs has not been effectively addressed, limiting their practical application and preventing them from fully exploiting the benefits of no post-processing, such as non-maximum suppression (NMS). In this paper, we first analyze the influence of NMS in modern real-time object detectors on inference speed, and establish an end-to-end speed benchmark. To avoid the inference delay caused by NMS, we propose a Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge. Specifically, we design an efficient hybrid encoder to efficiently process multi-scale features by decoupling the intra-scale interaction and cross-scale fusion, and propose IoU-aware query selection to improve the initialization of object queries. In addition, our proposed detector supports flexibly adjustment of the inference speed by using different decoder layers without the need for retraining, which facilitates the practical application of real-time object detectors. Our RT-DETR-L achieves 53.0% AP on COCO val2017 and 114 FPS on T4 GPU, while RT-DETR-X achieves 54.8% AP and 74 FPS, outperforming all YOLO detectors of the same scale in both speed and accuracy. Furthermore, our RT-DETR-R50 achieves 53.1% AP and 108 FPS, outperforming DINO-Deformable-DETR-R50 by 2.2% AP in accuracy and by about 21 times in FPS.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/rt_detr_overview.png"
alt="drawing" width="600"/>

<small> RT-DETR performance relative to YOLO models. Taken from the <a href="https://arxiv.org/abs/2304.08069">original paper.</a> </small>

The model version was contributed by [rafaelpadilla](https://huggingface.co/rafaelpadilla) and [sangbumchoi](https://github.com/SangbumChoi). The original code can be found [here](https://github.com/lyuwenyu/RT-DETR/).


## Usage tips

Initially, an image is processed using a pre-trained convolutional neural network, specifically a Resnet-D variant as referenced in the original code. This network extracts features from the final three layers of the architecture. Following this, a hybrid encoder is employed to convert the multi-scale features into a sequential array of image features. Then, a decoder, equipped with auxiliary prediction heads is used to refine the object queries. This process facilitates the direct generation of bounding boxes, eliminating the need for any additional post-processing to acquire the logits and coordinates for the bounding boxes.


key: _node_content
string_value:
{"id_": "5362b806-a334-4c05-8c9a-803ba88c0771", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14a0d8a7-c7fa-4d72-a4b1-005358c62fdb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19aa0f1993bb6090ee0e4a55d49e3671c5f2817609464a59dc3b7e8a89ca68ea", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9d545c8-1d0e-4188-af29-a4f964320f2e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "86b16eedcdaa07a3bf1d58ad9ecb2cc6ecd7cc4becdd1f7b10f3da1d21f8ecd0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd0a705c-4f7e-4ef3-b711-3b1ac491a8a1", "node_type": "1", "metadata": {}, "hash": "63d76cb9bd536d7c10a59dcb217146198c9293fd5cfc6d8daa9a51a5588c24e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2746, "end_char_idx": 5264, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips

Initially, an image is processed using a pre-trained convolutional neural network, specifically a Resnet-D variant as referenced in the original code. This network extracts features from the final three layers of the architecture. Following this, a hybrid encoder is employed to convert the multi-scale features into a sequential array of image features. Then, a decoder, equipped with auxiliary prediction heads is used to refine the object queries. This process facilitates the direct generation of bounding boxes, eliminating the need for any additional post-processing to acquire the logits and coordinates for the bounding boxes.

```py
>>> import torch
>>> import requests

>>> from PIL import Image
>>> from transformers import RTDetrForObjectDetection, RTDetrImageProcessor

>>> url = 'http://images.cocodataset.org/val2017/000000039769.jpg' 
>>> image = Image.open(requests.get(url, stream=True).raw)

>>> image_processor = RTDetrImageProcessor.from_pretrained("PekingU/rtdetr_r50vd")
>>> model = RTDetrForObjectDetection.from_pretrained("PekingU/rtdetr_r50vd")

>>> inputs = image_processor(images=image, return_tensors="pt")

>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> results = image_processor.post_process_object_detection(outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.3)

>>> for result in results:
...     for score, label_id, box in zip(result["scores"], result["labels"], result["boxes"]):
...         score, label = score.item(), label_id.item()
...         box = [round(i, 2) for i in box.tolist()]
...         print(f"{model.config.id2label[label]}: {score:.2f} {box}")
sofa: 0.97 [0.14, 0.38, 640.13, 476.21]
cat: 0.96 [343.38, 24.28, 640.14, 371.5]
cat: 0.96 [13.23, 54.18, 318.98, 472.22]
remote: 0.95 [40.11, 73.44, 175.96, 118.48]
remote: 0.92 [333.73, 76.58, 369.97, 186.99]
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with RT-DETR.

<PipelineTag pipeline="object-detection"/>

- Scripts for finetuning [`RTDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).
- Notebooks regarding inference and fine-tuning RT-DETR on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/RT-DETR).


key: _node_content
string_value:
{"id_": "fd0a705c-4f7e-4ef3-b711-3b1ac491a8a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14a0d8a7-c7fa-4d72-a4b1-005358c62fdb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "19aa0f1993bb6090ee0e4a55d49e3671c5f2817609464a59dc3b7e8a89ca68ea", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5362b806-a334-4c05-8c9a-803ba88c0771", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rt_detr/", "title": "RT-DETR", "tokens": 1617, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f3afb94a0b896475a8166d7905eedcda17d2ee1b10b907743254a547b9e479d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4737, "end_char_idx": 5685, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<PipelineTag pipeline="object-detection"/>

- Scripts for finetuning [`RTDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).
- Notebooks regarding inference and fine-tuning RT-DETR on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/RT-DETR). 🌎

## RTDetrConfig

[[autodoc]] RTDetrConfig

## RTDetrResNetConfig

[[autodoc]] RTDetrResNetConfig

## RTDetrImageProcessor

[[autodoc]] RTDetrImageProcessor
    - preprocess
    - post_process_object_detection

## RTDetrModel

[[autodoc]] RTDetrModel
    - forward

## RTDetrForObjectDetection

[[autodoc]] RTDetrForObjectDetection
    - forward

## RTDetrResNetBackbone

[[autodoc]] RTDetrResNetBackbone
    - forward


key: _node_content
string_value:
{"id_": "55dec6ad-8776-4500-bf68-4f8b2e7d19b2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d61f75e-2985-4b10-9426-c2a16771feec", "node_type": "1", "metadata": {}, "hash": "2e9c2619905eedbbd224c82c2e3dc0934d19acbf355c347fa2f378f811d987e6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3150, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Idefics2

## Overview

The Idefics2 model was proposed in [What matters when building vision-language models?](https://arxiv.org/abs/2405.02246) by Léo Tronchon, Hugo Laurencon, Victor Sanh. The accompanying blog post can be found [here](https://huggingface.co/blog/idefics2).

Idefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text
outputs. The model can answer questions about images, describe visual content, create stories grounded on multiple
images, or simply behave as a pure language model without visual inputs. It improves upon IDEFICS-1, notably on
document understanding, OCR, or visual reasoning. Idefics2 is lightweight (8 billion parameters) and treats
images in their native aspect ratio and resolution, which allows for varying inference efficiency.

The abstract from the paper is the following:

*The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/idefics2_architecture.png"
alt="drawing" width="600"/>

<small> Idefics2 architecture. Taken from the <a href="https://arxiv.org/abs/2405.02246">original paper.</a> </small>

This model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).
The original code can be found [here](https://huggingface.co/HuggingFaceM4/idefics2).

## Usage tips

- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.
- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.
- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.
- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.


key: _node_content
string_value:
{"id_": "1d61f75e-2985-4b10-9426-c2a16771feec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55dec6ad-8776-4500-bf68-4f8b2e7d19b2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4c2068202ab29adb4080a97a0c038b9636610559811e41ee66bbb41a09543ced", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea27d158-6cd4-429a-8fa3-31d1c8d2d9d2", "node_type": "1", "metadata": {}, "hash": "73f6cfa1b153a27e065e883bbd18df2da0e1c696748e6169faa538c2ce171e27", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1556, "end_char_idx": 3813, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/idefics2_architecture.png"
alt="drawing" width="600"/>

<small> Idefics2 architecture. Taken from the <a href="https://arxiv.org/abs/2405.02246">original paper.</a> </small>

This model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).
The original code can be found [here](https://huggingface.co/HuggingFaceM4/idefics2).

## Usage tips

- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.
- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.
- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.
- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.

Example of how to use the processor on chat messages:

```python
import requests
from PIL import Image
from transformers import Idefics2Processor, Idefics2ForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
url_2 = "http://images.cocodataset.org/val2017/000000219578.jpg"

image_1 = Image.open(requests.get(url_1, stream=True).raw)
image_2 = Image.open(requests.get(url_2, stream=True).raw)
images = [image_1, image_2]

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "What’s the difference between these two images?


key: _node_content
string_value:
{"id_": "ea27d158-6cd4-429a-8fa3-31d1c8d2d9d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d61f75e-2985-4b10-9426-c2a16771feec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e465749250c87c6aa4b0d166657a4155a88dd715d47b183bbeb4d7fc54d9e762", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "19caa2fc-08a0-4875-98ef-6290829b87fc", "node_type": "1", "metadata": {}, "hash": "b3466f9230102948534028cf5a078bdf96b5e6d0f73e2695f6dd430cbc1e5387", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2275, "end_char_idx": 4803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips

- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images in a batch for input to the model.
- The processor has a `do_image_splitting` option. If `True`, each input image will be split into 4 sub-images, and concatenated with the original to form 5 images. This is useful for increasing model performance. Make sure `processor.image_processor.do_image_splitting` is set to `False` if the model was not trained with this option.
- `text` passed to the processor should have the `<image>` tokens where the images should be inserted. And `<end_of_utterance>` at the end of each utterance if the text is a chat message.
- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as `text` to the processor.

Example of how to use the processor on chat messages:

```python
import requests
from PIL import Image
from transformers import Idefics2Processor, Idefics2ForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
url_2 = "http://images.cocodataset.org/val2017/000000219578.jpg"

image_1 = Image.open(requests.get(url_1, stream=True).raw)
image_2 = Image.open(requests.get(url_2, stream=True).raw)
images = [image_1, image_2]

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "What’s the difference between these two images?"},
        {"type": "image"},
        {"type": "image"},
    ],
}]

processor = Idefics2Processor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = Idefics2ForConditionalGeneration.from_pretrained("HuggingFaceM4/idefics2-8b")
model.to(device)

# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt
text = processor.apply_chat_template(messages, add_generation_prompt=True)
print(text)
# 'User: What’s the difference between these two images?<image><image><end_of_utterance>\nAssistant:'

inputs = processor(images=images, text=text, return_tensors="pt").to(device)

generated_text = model.generate(**inputs, max_new_tokens=500)
generated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]
print("Generated text:", generated_text)
```

- During training, it's important to determine which tokens the model should not learn. For Idefics2, this typically comes down to the image and padding tokens.


key: _node_content
string_value:
{"id_": "19caa2fc-08a0-4875-98ef-6290829b87fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea27d158-6cd4-429a-8fa3-31d1c8d2d9d2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dacdc0919ebc976e3c422c6036ffb3ec5d213a74a08e823fffac85621b1fa67a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cdaf5036-6716-4086-bc60-b3a7c0aa4cdb", "node_type": "1", "metadata": {}, "hash": "dd497a6a2c212fd436b639beda714e2a78ebf1d80259b40e5c353778051f3a4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3813, "end_char_idx": 6898, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
        {"type": "image"},
        {"type": "image"},
    ],
}]

processor = Idefics2Processor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = Idefics2ForConditionalGeneration.from_pretrained("HuggingFaceM4/idefics2-8b")
model.to(device)

# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt
text = processor.apply_chat_template(messages, add_generation_prompt=True)
print(text)
# 'User: What’s the difference between these two images?<image><image><end_of_utterance>\nAssistant:'

inputs = processor(images=images, text=text, return_tensors="pt").to(device)

generated_text = model.generate(**inputs, max_new_tokens=500)
generated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]
print("Generated text:", generated_text)
```

- During training, it's important to determine which tokens the model should not learn. For Idefics2, this typically comes down to the image and padding tokens. This means that one can create the labels as follows:

```python
import requests
from PIL import Image
from transformers import Idefics2Processor, Idefics2ForConditionalGeneration
import torch

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
url_2 = "http://images.cocodataset.org/val2017/000000219578.jpg"

image_1 = Image.open(requests.get(url_1, stream=True).raw)
image_2 = Image.open(requests.get(url_2, stream=True).raw)
images = [image_1, image_2]

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "What’s the difference between these two images?"},
        {"type": "image"},
        {"type": "image"},
    ],
},
{
    "role": "assistant",
    "content": [
        {"type": "text", "text": "The difference is that one image is about dogs and the other one about cats."},
    ],
}]

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Idefics2Processor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = Idefics2ForConditionalGeneration.from_pretrained("HuggingFaceM4/idefics2-8b")
model.to(device)

text = processor.apply_chat_template(messages, add_generation_prompt=False)
inputs = processor(images=images, text=text, return_tensors="pt").to(device)

labels = inputs.input_ids.clone()
labels[labels == processor.tokenizer.pad_token_id] = -100
labels[labels == model.config.image_token_id] = -100

inputs["labels"] = labels

outputs = model(**inputs)
loss = outputs.loss
loss.backward()
```

Do note that when training Idefics2 on multi-turn conversations between a user and an assistant, one typically also sets all the tokens corresponding to the user messages to -100.

## Model optimizations: Flash Attention

The code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.


key: _node_content
string_value:
{"id_": "cdaf5036-6716-4086-bc60-b3a7c0aa4cdb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "19caa2fc-08a0-4875-98ef-6290829b87fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "70e8f794498c53b2101a9bcf82f6e8ad2c4e61282a30b8201838bd44e7980cef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "09dc4b40-29ef-461b-b77d-167ef51097a2", "node_type": "1", "metadata": {}, "hash": "68725f6522432aad0a3c718574d3f68662557193856c8a429b8f6703178490a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5408, "end_char_idx": 8115, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
        {"type": "image"},
        {"type": "image"},
    ],
},
{
    "role": "assistant",
    "content": [
        {"type": "text", "text": "The difference is that one image is about dogs and the other one about cats."},
    ],
}]

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Idefics2Processor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = Idefics2ForConditionalGeneration.from_pretrained("HuggingFaceM4/idefics2-8b")
model.to(device)

text = processor.apply_chat_template(messages, add_generation_prompt=False)
inputs = processor(images=images, text=text, return_tensors="pt").to(device)

labels = inputs.input_ids.clone()
labels[labels == processor.tokenizer.pad_token_id] = -100
labels[labels == model.config.image_token_id] = -100

inputs["labels"] = labels

outputs = model(**inputs)
loss = outputs.loss
loss.backward()
```

Do note that when training Idefics2 on multi-turn conversations between a user and an assistant, one typically also sets all the tokens corresponding to the user messages to -100.

## Model optimizations: Flash Attention

The code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)

To load and run a model using Flash Attention-2, simply change the code snippet above with the following change:

```diff
model = Idefics2ForConditionalGeneration.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
+    torch_dtype=torch.float16,    
+    attn_implementation="flash_attention_2",
).to(device)
```

## Shrinking down Idefics2 using quantization

As the Idefics2 model has 8 billion parameters, that would require about 16GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), that requires only about 3.5GB of RAM.

Quantizing a model is as simple as passing a `quantization_config` to the model. One can change the code snippet above with the changes below.


key: _node_content
string_value:
{"id_": "09dc4b40-29ef-461b-b77d-167ef51097a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cdaf5036-6716-4086-bc60-b3a7c0aa4cdb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b52c943e259dd233add73ca0674bbdc89170363f787fb9f30534ecd53d5a0065", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9bb1c2ad-6e26-4efe-9518-481d345a1c84", "node_type": "1", "metadata": {}, "hash": "401b4134d593065e28611712b0303a7db7d356b3171682ca513cddf3b2048875", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6456, "end_char_idx": 9464, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Model optimizations: Flash Attention

The code snippets above showcase inference without any optimization tricks. However, one can drastically speed up the model by leveraging [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). Make also sure to load your model in half-precision (e.g. `torch.float16`)

To load and run a model using Flash Attention-2, simply change the code snippet above with the following change:

```diff
model = Idefics2ForConditionalGeneration.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
+    torch_dtype=torch.float16,    
+    attn_implementation="flash_attention_2",
).to(device)
```

## Shrinking down Idefics2 using quantization

As the Idefics2 model has 8 billion parameters, that would require about 16GB of GPU RAM in half precision (float16), since each parameter is stored in 2 bytes. However, one can shrink down the size of the model using [quantization](../quantization.md). If the model is quantized to 4 bits (or half a byte per parameter), that requires only about 3.5GB of RAM.

Quantizing a model is as simple as passing a `quantization_config` to the model. One can change the code snippet above with the changes below. We'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):

```diff
+ from transformers import BitsAndBytesConfig

+ quantization_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_compute_dtype=torch.float16
+ )
model = Idefics2ForConditionalGeneration.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
+    torch_dtype=torch.float16,    
+    quantization_config=quantization_config,
).to(device)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Idefics2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A notebook on how to fine-tune Idefics2 on a custom dataset using the [Trainer](../main_classes/trainer.md) can be found [here](https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing). It supports both full fine-tuning as well as (quantized) LoRa.
- A script regarding how to fine-tune Idefics2 using the TRL library can be found [here](https://gist.github.com/edbeeching/228652fc6c2b29a1641be5a5778223cb).


key: _node_content
string_value:
{"id_": "9bb1c2ad-6e26-4efe-9518-481d345a1c84", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f5938c0-e81c-4ef3-9943-6596d7b9625d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bbfb9ca238891a9e7786811f3042c65b852c284c1896b39025b97d3e39a6df91", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "09dc4b40-29ef-461b-b77d-167ef51097a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics2/", "title": "Idefics2", "tokens": 2631, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8dcafc1f7006df62b9d7b2937366528bbf5e0fcf4610526922ed82e6e6b84b3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8054, "end_char_idx": 9992, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
One can change the code snippet above with the changes below. We'll leverage the BitsAndyBytes quantization (but refer to [this page](../quantization.md) for other quantization methods):

```diff
+ from transformers import BitsAndBytesConfig

+ quantization_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_compute_dtype=torch.float16
+ )
model = Idefics2ForConditionalGeneration.from_pretrained(
    "HuggingFaceM4/idefics2-8b",
+    torch_dtype=torch.float16,    
+    quantization_config=quantization_config,
).to(device)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Idefics2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A notebook on how to fine-tune Idefics2 on a custom dataset using the [Trainer](../main_classes/trainer.md) can be found [here](https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing). It supports both full fine-tuning as well as (quantized) LoRa.
- A script regarding how to fine-tune Idefics2 using the TRL library can be found [here](https://gist.github.com/edbeeching/228652fc6c2b29a1641be5a5778223cb).
- Demo notebook regarding fine-tuning Idefics2 for JSON extraction use cases can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Idefics2). 🌎

## Idefics2Config

[[autodoc]] Idefics2Config


## Idefics2Model

[[autodoc]] Idefics2Model
    - forward


## Idefics2ForConditionalGeneration

[[autodoc]] Idefics2ForConditionalGeneration
    - forward


## Idefics2ImageProcessor
[[autodoc]] Idefics2ImageProcessor
    - preprocess


## Idefics2Processor
[[autodoc]] Idefics2Processor
    - __call__


key: _node_content
string_value:
{"id_": "692c8266-7336-48c3-913d-afe271594179", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/informer/", "title": "Informer", "tokens": 782, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1a260969-de5e-4b58-a466-c56824d3aa3a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/informer/", "title": "Informer", "tokens": 782, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8ba2ab01f0906f383d41bc0bd5334a503bbe4a54112568872d1df5764b38303e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Informer

## Overview

The Informer model was proposed in [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://arxiv.org/abs/2012.07436) by Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.

This method introduces a Probabilistic Attention mechanism to select the "active" queries rather than the "lazy" queries and provides a sparse Transformer thus mitigating the quadratic compute and memory requirements of vanilla attention.

The abstract from the paper is the following:

*Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L logL) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.*

This model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif).
The original code can be found [here](https://github.com/zhouhaoyi/Informer2020).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- Check out the Informer blog-post in HuggingFace blog: [Multivariate Probabilistic Time Series Forecasting with Informer](https://huggingface.co/blog/informer)

## InformerConfig

[[autodoc]] InformerConfig

## InformerModel

[[autodoc]] InformerModel
    - forward

## InformerForPrediction

[[autodoc]] InformerForPrediction
    - forward


key: _node_content
string_value:
{"id_": "e8c2a231-7ed6-4032-be2d-ec0dac84758c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/herbert/", "title": "HerBERT", "tokens": 804, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4ee53d7b-b912-4476-8424-cee5ce97666e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/herbert/", "title": "HerBERT", "tokens": 804, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "505b4aa54540e252ba66add327ff72547b8f61f3ca665815ac02908c35c0ff7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# HerBERT

## Overview

The HerBERT model was proposed in [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://www.aclweb.org/anthology/2020.acl-main.111.pdf) by Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and
Ireneusz Gawlik. It is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic
masking of whole words.

The abstract from the paper is the following:

*In recent years, a series of Transformer-based models unlocked major improvements in general natural language
understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which
allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of
languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language
understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing
datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new
sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and
promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and
applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language,
which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an
extensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based
models.*

This model was contributed by [rmroczkowski](https://huggingface.co/rmroczkowski). The original code can be found
[here](https://github.com/allegro/HerBERT).


## Usage example

```python
>>> from transformers import HerbertTokenizer, RobertaModel

>>> tokenizer = HerbertTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
>>> model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")

>>> encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors="pt")
>>> outputs = model(encoded_input)

>>> # HerBERT can also be loaded using AutoTokenizer and AutoModel:
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
>>> model = AutoModel.from_pretrained("allegro/herbert-klej-cased-v1")
```

<Tip>

Herbert implementation is the same as `BERT` except for the tokenization method. Refer to [BERT documentation](bert) 
for API reference and examples.  

</Tip>

## HerbertTokenizer

[[autodoc]] HerbertTokenizer

## HerbertTokenizerFast

[[autodoc]] HerbertTokenizerFast


key: _node_content
string_value:
{"id_": "5f783ce3-af8d-4be7-99bd-3625fad71f32", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encodec/", "title": "EnCodec", "tokens": 859, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79a2a2f9-da01-45a4-9762-78680a786a1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/encodec/", "title": "EnCodec", "tokens": 859, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1ca24f674aea87c36ed13690848e5a275c04c16a0fb5167f7d74feac3ac3a881", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3127, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# EnCodec

## Overview

The EnCodec neural codec model was proposed in [High Fidelity Neural Audio Compression](https://arxiv.org/abs/2210.13438) by Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi.

The abstract from the paper is the following:

*We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio.*

This model was contributed by [Matthijs](https://huggingface.co/Matthijs), [Patrick Von Platen](https://huggingface.co/patrickvonplaten) and [Arthur Zucker](https://huggingface.co/ArthurZ). 
The original code can be found [here](https://github.com/facebookresearch/encodec).

## Usage example 

Here is a quick example of how to encode and decode an audio using this model:

```python 
>>> from datasets import load_dataset, Audio
>>> from transformers import EncodecModel, AutoProcessor
>>> librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

>>> model = EncodecModel.from_pretrained("facebook/encodec_24khz")
>>> processor = AutoProcessor.from_pretrained("facebook/encodec_24khz")
>>> librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=processor.sampling_rate))
>>> audio_sample = librispeech_dummy[-1]["audio"]["array"]
>>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors="pt")

>>> encoder_outputs = model.encode(inputs["input_values"], inputs["padding_mask"])
>>> audio_values = model.decode(encoder_outputs.audio_codes, encoder_outputs.audio_scales, inputs["padding_mask"])[0]
>>> # or the equivalent with a forward pass
>>> audio_values = model(inputs["input_values"], inputs["padding_mask"]).audio_values
```

## EncodecConfig

[[autodoc]] EncodecConfig

## EncodecFeatureExtractor

[[autodoc]] EncodecFeatureExtractor
    - __call__

## EncodecModel

[[autodoc]] EncodecModel
    - decode
    - encode
    - forward


key: _node_content
string_value:
{"id_": "ae016fcb-f0d9-4099-a111-20559288ac36", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip-2/", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a2c0cc58-0225-44c2-ad28-325f530264b2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip-2/", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "be1d4651acbfadf371ebb50073dcaf270ddbd245a475b77e33181702d8ac20a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "653f7908-3f48-4c14-93e3-73f9db4e50e2", "node_type": "1", "metadata": {}, "hash": "4ccfce4e31e27ed8ce348277caa67d75365ca27819f56425c13f094d5110ccba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3084, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BLIP-2

## Overview

The BLIP-2 model was proposed in [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) by
Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. BLIP-2 leverages frozen pre-trained image encoders and large language models (LLMs) by training a lightweight, 12-layer Transformer
encoder in between them, achieving state-of-the-art performance on various vision-language tasks. Most notably, BLIP-2 improves upon [Flamingo](https://arxiv.org/abs/2204.14198), an 80 billion parameter model, by 8.7%
on zero-shot VQAv2 with 54x fewer trainable parameters. 

The abstract from the paper is the following:

*The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg"
alt="drawing" width="600"/> 

<small> BLIP-2 architecture. Taken from the <a href="https://arxiv.org/abs/2301.12597">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).

## Usage tips

- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.
- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BLIP-2.

- Demo notebooks for BLIP-2 for image captioning, visual question answering (VQA) and chat-like conversations can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "653f7908-3f48-4c14-93e3-73f9db4e50e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip-2/", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a2c0cc58-0225-44c2-ad28-325f530264b2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip-2/", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "be1d4651acbfadf371ebb50073dcaf270ddbd245a475b77e33181702d8ac20a8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae016fcb-f0d9-4099-a111-20559288ac36", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip-2/", "title": "BLIP-2", "tokens": 1084, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7d8c8edfc7364653241cae2c8e1ee86451746ccf5cc3138308b4459ac5fa886f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1502, "end_char_idx": 3686, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg"
alt="drawing" width="600"/> 

<small> BLIP-2 architecture. Taken from the <a href="https://arxiv.org/abs/2301.12597">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/salesforce/LAVIS/tree/5ee63d688ba4cebff63acee04adaef2dee9af207).

## Usage tips

- BLIP-2 can be used for conditional text generation given an image and an optional text prompt. At inference time, it's recommended to use the [`generate`] method.
- One can use [`Blip2Processor`] to prepare images for the model, and decode the predicted tokens ID's back to text.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BLIP-2.

- Demo notebooks for BLIP-2 for image captioning, visual question answering (VQA) and chat-like conversations can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## Blip2Config

[[autodoc]] Blip2Config
    - from_vision_qformer_text_configs

## Blip2VisionConfig

[[autodoc]] Blip2VisionConfig

## Blip2QFormerConfig

[[autodoc]] Blip2QFormerConfig

## Blip2Processor

[[autodoc]] Blip2Processor

## Blip2VisionModel

[[autodoc]] Blip2VisionModel
    - forward

## Blip2QFormerModel

[[autodoc]] Blip2QFormerModel
    - forward

## Blip2Model

[[autodoc]] Blip2Model
    - forward
    - get_text_features
    - get_image_features
    - get_qformer_features

## Blip2ForConditionalGeneration

[[autodoc]] Blip2ForConditionalGeneration
    - forward
    - generate


key: _node_content
string_value:
{"id_": "0922b9e9-8407-47f9-8338-8009d43a50c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a5558122-398e-47cb-8128-e6ca8329335f", "node_type": "1", "metadata": {}, "hash": "88eaddb12c295fb028a6679b546f5d8db9e7856402de575dc51d1a2bc812b315", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3295, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BERT

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=bert">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-bert-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/bert-base-uncased">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The BERT model was proposed in [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. It's a
bidirectional transformer pretrained using a combination of masked language modeling objective and next sentence
prediction on a large corpus comprising the Toronto Book Corpus and Wikipedia.

The abstract from the paper is the following:

*We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations
from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional
representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result,
the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models
for a wide range of tasks, such as question answering and language inference, without substantial task-specific
architecture modifications.*

*BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural
language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI
accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute
improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).

## Usage tips

- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is
  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.
- Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:

    * a special mask token with probability 0.8
    * a random token different from the one masked with probability 0.1
    * the same token with probability 0.1
    
- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use.


key: _node_content
string_value:
{"id_": "a5558122-398e-47cb-8128-e6ca8329335f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0922b9e9-8407-47f9-8338-8009d43a50c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b7b9ec4cbe73d13edcc74dbbf1fcbcd5c283ba2d420b1dcad805a5542ed056ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dec65086-69b2-40f4-8299-848cca9a9863", "node_type": "1", "metadata": {}, "hash": "56260d25965577b9debcd773320033151e7c2d60124058c3f914b3dc667d34f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1872, "end_char_idx": 4217, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/google-research/bert).

## Usage tips

- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is
  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.
- Corrupts the inputs by using random masking, more precisely, during pretraining, a given percentage of tokens (usually 15%) is masked by:

    * a special mask token with probability 0.8
    * a random token different from the one masked with probability 0.1
    * the same token with probability 0.1
    
- The model must predict the original sentence, but has a second objective: inputs are two sentences A and B (with a separation token in between). With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-uncased", torch_dtype=torch.float16, attn_implementation="sdpa")
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-80GB, CPUx12, RAM 96.6GB, PyTorch 2.2.0, OS Ubuntu 22.04) with `float16`, we saw the 
following speedups during training and inference.


key: _node_content
string_value:
{"id_": "dec65086-69b2-40f4-8299-848cca9a9863", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5558122-398e-47cb-8128-e6ca8329335f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3dd74e60192dfa7a5107e37c53392ae86e8b2b3035b1eda4700bfd1ab9e3d6e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0db5c08b-78d5-4751-b8b9-91dea1e080b4", "node_type": "1", "metadata": {}, "hash": "1c69cf33c2f8a695fe3d2039c85992cb4141057bb08c52c9070e5ed3d540d4e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2847, "end_char_idx": 6418, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With probability 50%, the sentences are consecutive in the corpus, in the remaining 50% they are not related. The model has to predict if the sentences are consecutive or not.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import BertModel

model = BertModel.from_pretrained("bert-base-uncased", torch_dtype=torch.float16, attn_implementation="sdpa")
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-80GB, CPUx12, RAM 96.6GB, PyTorch 2.2.0, OS Ubuntu 22.04) with `float16`, we saw the 
following speedups during training and inference.

#### Training

|batch_size|seq_len|Time per batch (eager - s)|Time per batch (sdpa - s)|Speedup (%)|Eager peak mem (MB)|sdpa peak mem (MB)|Mem saving (%)|
|----------|-------|--------------------------|-------------------------|-----------|-------------------|------------------|--------------|
|4         |256    |0.023                     |0.017                    |35.472     |939.213            |764.834           |22.800        |
|4         |512    |0.023                     |0.018                    |23.687     |1970.447           |1227.162          |60.569        |
|8         |256    |0.023                     |0.018                    |23.491     |1594.295           |1226.114          |30.028        |
|8         |512    |0.035                     |0.025                    |43.058     |3629.401           |2134.262          |70.054        |
|16        |256    |0.030                     |0.024                    |25.583     |2874.426           |2134.262          |34.680        |
|16        |512    |0.064                     |0.044                    |46.223     |6964.659           |3961.013          |75.830        |

#### Inference

|batch_size|seq_len|Per token latency eager (ms)|Per token latency SDPA (ms)|Speedup (%)|Mem eager (MB)|Mem BT (MB)|Mem saved (%)|
|----------|-------|----------------------------|---------------------------|-----------|--------------|-----------|-------------|
|1         |128    |5.736                       |4.987                      |15.022     |282.661       |282.924    |-0.093       |
|1         |256    |5.689                       |4.945                      |15.055     |298.686       |298.948    |-0.088       |
|2         |128    |6.154                       |4.982                      |23.521     |314.523       |314.785    |-0.083       |
|2         |256    |6.201                       |4.949                      |25.303     |347.546       |347.033    |0.148        |
|4         |128    |6.049                       |4.987                      |21.305     |378.895       |379.301    |-0.107       |
|4         |256    |6.285                       |5.364                      |17.166     |443.209       |444.382    |-0.264       |


key: _node_content
string_value:
{"id_": "0db5c08b-78d5-4751-b8b9-91dea1e080b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dec65086-69b2-40f4-8299-848cca9a9863", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5fa313f40223e8a841a77ede99e4069188011bde968924c4c95be9e79ef0cf56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3f5905e-e576-4ffb-8cea-d21c028fb0ef", "node_type": "1", "metadata": {}, "hash": "def6929d6cb1a9aa7ee8e3c4dd1570112e269ee633a6aeea9ca0681cd47bc9f3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6422, "end_char_idx": 9651, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on [BERT Text Classification in a different language](https://www.philschmid.de/bert-text-classification-in-a-different-language).
- A notebook for [Finetuning BERT (and friends) for multi-label text classification](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb).
- A notebook on how to [Finetune BERT for multi-label classification using PyTorch](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb). 🌎
- A notebook on how to [warm-start an EncoderDecoder model with BERT for summarization](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb).
- [`BertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- A blog post on how to use [Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition](https://www.philschmid.de/huggingface-transformers-keras-tf).
- A notebook for [Finetuning BERT for named-entity recognition](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb) using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this [version](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT.ipynb) of the notebook instead.
- [`BertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).


key: _node_content
string_value:
{"id_": "e3f5905e-e576-4ffb-8cea-d21c028fb0ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0db5c08b-78d5-4751-b8b9-91dea1e080b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "389375399a9673c573ab5087e4dbf21a087a72118f0133ead32ac41e90a75d18", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d2d02bd-2374-43df-891b-885c7080b115", "node_type": "1", "metadata": {}, "hash": "e299b0ad534f982b3b59f1e84cbb864c29c47248df5ffc5095462e6c0bc0f70a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8233, "end_char_idx": 11372, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxBertForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- A blog post on how to use [Hugging Face Transformers with Keras: Fine-tune a non-English BERT for Named Entity Recognition](https://www.philschmid.de/huggingface-transformers-keras-tf).
- A notebook for [Finetuning BERT for named-entity recognition](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT_only_first_wordpiece.ipynb) using only the first wordpiece of each word in the word label during tokenization. To propagate the label of the word to all wordpieces, see this [version](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Custom_Named_Entity_Recognition_with_BERT.ipynb) of the notebook instead.
- [`BertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- [`BertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.


key: _node_content
string_value:
{"id_": "7d2d02bd-2374-43df-891b-885c7080b115", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3f5905e-e576-4ffb-8cea-d21c028fb0ef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "56d43b3c18d259c073506e30b724515cf3527c22f423e35c3d43a33dc7037f90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "354d8335-78a1-4c7e-8739-32e6f21f2d16", "node_type": "1", "metadata": {}, "hash": "256a523c841795735baf57a8e6e33e7916b89edf7ca973b48a589892b04ddd3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9950, "end_char_idx": 12983, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxBertForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- [`BertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxBertForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`BertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**
- [`BertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).


key: _node_content
string_value:
{"id_": "354d8335-78a1-4c7e-8739-32e6f21f2d16", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc94e34f-b3f8-4595-90e5-28882d743fe2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "134232600056dbbfa63426804f3873813c64bc37046c0a96955261c5b34ded69", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7d2d02bd-2374-43df-891b-885c7080b115", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bert/", "title": "BERT", "tokens": 4596, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c1123de2a1d532531a730b977769d43277b4da08f488063bd0ba9981ea745ebd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbddecf0-34c9-4ffb-852f-18a3367c5bb8", "node_type": "1", "metadata": {}, "hash": "f2a867e9151f9196b3812d1ed74925fbe0495840bdfaee09ab74281f1e07f7b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11373, "end_char_idx": 14364, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`BertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxBertForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**
- [`BertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFBertForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

⚡️ **Inference**
- A blog post on how to [Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia](https://huggingface.co/blog/bert-inferentia-sagemaker).
- A blog post on how to [Accelerate BERT inference with DeepSpeed-Inference on GPUs](https://www.philschmid.de/bert-deepspeed-inference).

⚙️ **Pretraining**
- A blog post on [Pre-Training BERT with Hugging Face Transformers and Habana Gaudi](https://www.philschmid.de/pre-training-bert-habana).

🚀 **Deploy**
- A blog post on how to [Convert Transformers to ONNX with Hugging Face Optimum](https://www.philschmid.de/convert-transformers-to-onnx).
- A blog post on how to [Setup Deep Learning environment for Hugging Face Transformers with Habana Gaudi on AWS](https://www.philschmid.de/getting-started-habana-gaudi#conclusion).
- A blog post on [Autoscaling BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module](https://www.philschmid.de/terraform-huggingface-amazon-sagemaker-advanced).
- A blog post on [Serverless BERT with HuggingFace, AWS Lambda, and Docker](https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker).
- A blog post on [Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler](https://www.philschmid.de/huggingface-amazon-sagemaker-training-compiler).


