# embedding_metadata

key: _node_content
string_value:
{"id_": "7d119e95-82ad-4401-98fa-a55cc50b1020", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0dd02a0-dbdc-48a8-a986-7cde833c91ba", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "8b008555e541080dd81bad44cd27b474127daf556111cf7282721a29acfc2f31", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b2e5258-421e-4b48-96b6-9dab4aad39be", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt2", "title": "OpenAI GPT2", "tokens": 4429, "retrieve_doc": true, "source": "transformers"}, "hash": "31a8c05e1b50dbab2ffd6f7bf13fd5daf460ba68bab2286030b39bf750933653", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12774, "end_char_idx": 15876, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🌎
- A notebook on how to [finetune GPT2 to generate tweets in the style of your favorite Twitter user](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb). 🌎
- [Causal language modeling](https://huggingface.co/course/en/chapter7/6?fw=pt#training-a-causal-language-model-from-scratch) chapter of the 🤗 Hugging Face Course.
- [`GPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling), [text generation example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation), and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_clmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxGPT2LMHeadModel`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/causal_language_modeling_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## GPT2Config

[[autodoc]] GPT2Config

## GPT2Tokenizer

[[autodoc]] GPT2Tokenizer
    - save_vocabulary

## GPT2TokenizerFast

[[autodoc]] GPT2TokenizerFast

## GPT2 specific outputs

[[autodoc]] models.gpt2.modeling_gpt2.GPT2DoubleHeadsModelOutput

[[autodoc]] models.gpt2.modeling_tf_gpt2.TFGPT2DoubleHeadsModelOutput

<frameworkcontent>
<pt>

## GPT2Model

[[autodoc]] GPT2Model
    - forward

## GPT2LMHeadModel

[[autodoc]] GPT2LMHeadModel
    - forward

## GPT2DoubleHeadsModel

[[autodoc]] GPT2DoubleHeadsModel
    - forward

## GPT2ForQuestionAnswering

[[autodoc]] GPT2ForQuestionAnswering
    - forward

## GPT2ForSequenceClassification

[[autodoc]] GPT2ForSequenceClassification
    - forward

## GPT2ForTokenClassification

[[autodoc]] GPT2ForTokenClassification
    - forward

</pt>
<tf>

## TFGPT2Model

[[autodoc]] TFGPT2Model
    - call

## TFGPT2LMHeadModel

[[autodoc]] TFGPT2LMHeadModel
    - call

## TFGPT2DoubleHeadsModel

[[autodoc]] TFGPT2DoubleHeadsModel
    - call

## TFGPT2ForSequenceClassification

[[autodoc]] TFGPT2ForSequenceClassification
    - call

## TFSequenceClassifierOutputWithPast

[[autodoc]] modeling_tf_outputs.TFSequenceClassifierOutputWithPast

## TFGPT2Tokenizer

[[autodoc]] TFGPT2Tokenizer

</tf>
<jax>

## FlaxGPT2Model

[[autodoc]] FlaxGPT2Model
    - __call__

## FlaxGPT2LMHeadModel

[[autodoc]] FlaxGPT2LMHeadModel
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "78165be4-0f26-4fd1-b7ea-dd69ab44d1a8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83c12a25-2781-4576-88cd-9e64bf041e7d", "node_type": "1", "metadata": {}, "hash": "97d15610b46e52c22c5aef59a729fd3425ebfbce03423e9ce4f3efa9ef32b83f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BROS

## Overview

The BROS model was proposed in [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://arxiv.org/abs/2108.04539) by Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, Sungrae Park.

BROS stands for *BERT Relying On Spatiality*. It is an encoder-only Transformer model that takes a sequence of tokens and their bounding boxes as inputs and outputs a sequence of hidden states. BROS encode relative spatial information instead of using absolute spatial information.

It is pre-trained with two objectives: a token-masked language modeling objective (TMLM) used in BERT, and a novel area-masked language modeling objective (AMLM)
In TMLM, tokens are randomly masked, and the model predicts the masked tokens using spatial information and other unmasked tokens.
AMLM is a 2D version of TMLM. It randomly masks text tokens and predicts with the same information as TMLM, but it masks text blocks (areas).

`BrosForTokenClassification` has a simple linear layer on top of BrosModel. It predicts the label of each token.
`BrosSpadeEEForTokenClassification` has an `initial_token_classifier` and `subsequent_token_classifier` on top of BrosModel. `initial_token_classifier` is used to predict the first token of each entity, and `subsequent_token_classifier` is used to predict the next token of within entity. `BrosSpadeELForTokenClassification` has an `entity_linker` on top of BrosModel. `entity_linker` is used to predict the relation between two entities.

`BrosForTokenClassification` and `BrosSpadeEEForTokenClassification` essentially perform the same job. However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.

`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.

BROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.

The abstract from the paper is the following:

*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.


key: _node_content
string_value:
{"id_": "83c12a25-2781-4576-88cd-9e64bf041e7d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78165be4-0f26-4fd1-b7ea-dd69ab44d1a8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "0b5d6965c6205045e767e448d585a8eb2337ebc11d893b94e68ced61dfcca0ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cdd83f90-9407-440f-8267-bfc883e77d93", "node_type": "1", "metadata": {}, "hash": "a51d28b889aa12b15039a238fa0681ba8d697c6c02ef2efef96e03139551ee02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1663, "end_char_idx": 4829, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, `BrosForTokenClassification` assumes input tokens are perfectly serialized (which is very challenging task since they exist in a 2D space), while `BrosSpadeEEForTokenClassification` allows for more flexibility in handling serialization errors as it predicts next connection tokens from one token.

`BrosSpadeELForTokenClassification` perform the intra-entity linking task. It predicts relation from one token (of one entity) to another token (of another entity) if these two entities share some relation.

BROS achieves comparable or better result on Key Information Extraction (KIE) benchmarks such as FUNSD, SROIE, CORD and SciTSR, without relying on explicit visual features.

The abstract from the paper is the following:

*Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*

This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).

## Usage tips and examples

- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.

```python
def expand_and_normalize_bbox(bboxes, doc_width, doc_height):
    # here, bboxes are numpy array

    # Normalize bbox -> 0 ~ 1
    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width
    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height
```

- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,


key: _node_content
string_value:
{"id_": "cdd83f90-9407-440f-8267-bfc883e77d93", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b10a5617-f0fa-4f52-9f3b-0fc1dff0b98c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "bd7fb45aaab4c49a95f49dd399e04059e923c316980c3363bdbb76bcc3326d65", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83c12a25-2781-4576-88cd-9e64bf041e7d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bros", "title": "BROS", "tokens": 1607, "retrieve_doc": true, "source": "transformers"}, "hash": "0a666a59f58a765d86aa516e27753bd154e4422ea8b740ffe0422bd69eea40be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3298, "end_char_idx": 6382, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This paper also reveals two real-world challenges in KIE tasks-(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples-and demonstrates the superiority of BROS over previous methods.*

This model was contributed by [jinho8345](https://huggingface.co/jinho8345). The original code can be found [here](https://github.com/clovaai/bros).

## Usage tips and examples

- [`~transformers.BrosModel.forward`] requires `input_ids` and `bbox` (bounding box). Each bounding box should be in (x0, y0, x1, y1) format (top-left corner, bottom-right corner). Obtaining of Bounding boxes depends on external OCR system. The `x` coordinate should be normalized by document image width, and the `y` coordinate should be normalized by document image height.

```python
def expand_and_normalize_bbox(bboxes, doc_width, doc_height):
    # here, bboxes are numpy array

    # Normalize bbox -> 0 ~ 1
    bboxes[:, [0, 2]] = bboxes[:, [0, 2]] / width
    bboxes[:, [1, 3]] = bboxes[:, [1, 3]] / height
```

- [`~transformers.BrosForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`, `~transformers.BrosSpadeEEForTokenClassification.forward`] require not only `input_ids` and `bbox` but also `box_first_token_mask` for loss calculation. It is a mask to filter out non-first tokens of each box. You can obtain this mask by saving start token indices of bounding boxes when creating `input_ids` from words. You can make `box_first_token_mask` with following code,


```python
def make_box_first_token_mask(bboxes, words, tokenizer, max_seq_length=512):

    box_first_token_mask = np.zeros(max_seq_length, dtype=np.bool_)

    # encode(tokenize) each word from words (List[str])
    input_ids_list: List[List[int]] = [tokenizer.encode(e, add_special_tokens=False) for e in words]

    # get the length of each box
    tokens_length_list: List[int] = [len(l) for l in input_ids_list]

    box_end_token_indices = np.array(list(itertools.accumulate(tokens_length_list)))
    box_start_token_indices = box_end_token_indices - np.array(tokens_length_list)

    # filter out the indices that are out of max_seq_length
    box_end_token_indices = box_end_token_indices[box_end_token_indices < max_seq_length - 1]
    if len(box_start_token_indices) > len(box_end_token_indices):
        box_start_token_indices = box_start_token_indices[: len(box_end_token_indices)]

    # set box_start_token_indices to True
    box_first_token_mask[box_start_token_indices] = True

    return box_first_token_mask

```

## Resources

- Demo scripts can be found [here](https://github.com/clovaai/bros).

## BrosConfig

[[autodoc]] BrosConfig

## BrosProcessor

[[autodoc]] BrosProcessor
    - __call__

## BrosModel

[[autodoc]] BrosModel
    - forward


## BrosForTokenClassification

[[autodoc]] BrosForTokenClassification
    - forward

## BrosSpadeEEForTokenClassification

[[autodoc]] BrosSpadeEEForTokenClassification
    - forward

## BrosSpadeELForTokenClassification

[[autodoc]] BrosSpadeELForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "926fa27b-f07b-4b7e-b12f-e110c50685d6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f19435bd-da59-4db2-a96e-d489189cb136", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "365a6391-73e6-4a1c-bc53-618b94093636", "node_type": "1", "metadata": {}, "hash": "7e9a9bea7f2e43028c15abf364e8ff9680e7e382d6c64fa6eb7e1b10a403b0dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3006, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoBERTa

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=roberta">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-roberta-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/roberta-base">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
<a href="https://huggingface.co/papers/1907.11692">
<img alt="Paper page" src="https://img.shields.io/badge/Paper%20page-1907.11692-green">
</a>
</div>

## Overview

The RoBERTa model was proposed in [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, [Myle Ott](https://huggingface.co/myleott), Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018.

It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with
much larger mini-batches and learning rates.

The abstract from the paper is the following:

*Language model pretraining has led to significant performance gains but careful comparison between different
approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes,
and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication
study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and
training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every
model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results
highlight the importance of previously overlooked design choices, and raise questions about the source of recently
reported improvements. We release our models and code.*

This model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).

## Usage tips

- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup
  for RoBERTa pretrained models.
- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
  different pretraining scheme.
- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just
  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).
- RoBERTa is similar to BERT but with better pretraining techniques:

    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.
    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).
    * Larger batches: Training uses larger batches.


key: _node_content
string_value:
{"id_": "365a6391-73e6-4a1c-bc53-618b94093636", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f19435bd-da59-4db2-a96e-d489189cb136", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "926fa27b-f07b-4b7e-b12f-e110c50685d6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "f919e9d84035c7557106fe58a50fae2bb5596a128f5914b320741e3d14102c87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50b1bcaa-256f-4bcd-83d3-a9f67289daf2", "node_type": "1", "metadata": {}, "hash": "795a632c853f71615069acd4102ba841a2189e49b056d2a7ce1b461bb4899719", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1411, "end_char_idx": 4760, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We present a replication
study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and
training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every
model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results
highlight the importance of previously overlooked design choices, and raise questions about the source of recently
reported improvements. We release our models and code.*

This model was contributed by [julien-c](https://huggingface.co/julien-c). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/roberta).

## Usage tips

- This implementation is the same as [`BertModel`] with a minor tweak to the embeddings, as well as a setup
  for RoBERTa pretrained models.
- RoBERTa has the same architecture as BERT but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
  different pretraining scheme.
- RoBERTa doesn't have `token_type_ids`, so you don't need to indicate which token belongs to which segment. Just
  separate your segments with the separation token `tokenizer.sep_token` (or `</s>`).
- RoBERTa is similar to BERT but with better pretraining techniques:

    * Dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all.
    * Sentence packing: Sentences are packed together to reach 512 tokens (so the sentences are in an order that may span several documents).
    * Larger batches: Training uses larger batches.
    * Byte-level BPE vocabulary: Uses BPE with bytes as a subunit instead of characters, accommodating Unicode characters.
- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).
- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.
- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). 🌎
- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).


key: _node_content
string_value:
{"id_": "50b1bcaa-256f-4bcd-83d3-a9f67289daf2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f19435bd-da59-4db2-a96e-d489189cb136", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "365a6391-73e6-4a1c-bc53-618b94093636", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5fcf0966c0c2522aa49518d5dec924edefff4264ba581f7f50e961b2fe5c6faa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "538f3a57-0878-4ff9-a7b8-4eec740b866c", "node_type": "1", "metadata": {}, "hash": "dff54f4b19bb58959515f781b6e03fca32540dc10c7c42313b3d29bfd90cf894", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3130, "end_char_idx": 6308, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [CamemBERT](camembert) is a wrapper around RoBERTa. Refer to its model page for usage examples.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog on [Getting Started with Sentiment Analysis on Twitter](https://huggingface.co/blog/sentiment-analysis-twitter) using RoBERTa and the [Inference API](https://huggingface.co/inference-api).
- A blog on [Opinion Classification with Kili and Hugging Face AutoTrain](https://huggingface.co/blog/opinion-classification-with-kili) using RoBERTa.
- A notebook on how to [finetune RoBERTa for sentiment analysis](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb). 🌎
- [`RobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa.


key: _node_content
string_value:
{"id_": "538f3a57-0878-4ff9-a7b8-4eec740b866c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f19435bd-da59-4db2-a96e-d489189cb136", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50b1bcaa-256f-4bcd-83d3-a9f67289daf2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "1c8edd7cd0038c9ff98768bf24ce02978a963097e08e9033a1973f66eea6b5ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0bebc9b3-9b7e-4769-8345-ead090e702b8", "node_type": "1", "metadata": {}, "hash": "778595b00b96a9a20dc44777b666f8f07b8ce664384bef41d5348caabfbd1321", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4761, "end_char_idx": 7927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`RobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="fill-mask"/>

- A blog on [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) with RoBERTa.
- [`RobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.
- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).


key: _node_content
string_value:
{"id_": "0bebc9b3-9b7e-4769-8345-ead090e702b8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f19435bd-da59-4db2-a96e-d489189cb136", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "538f3a57-0878-4ff9-a7b8-4eec740b866c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "e4ee4ab04e6d3ac9e114bfae173a82207e06a76022c96955f213bfba316f3d50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5089f9d-e331-49f2-a178-0f7ea5baa6f2", "node_type": "1", "metadata": {}, "hash": "d3f3a2e5f89a5f00130aaf662421b136792d37673c33ead1919633573853632d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6636, "end_char_idx": 9144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling task guide](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- A blog on [Accelerated Inference with Optimum and Transformers Pipelines](https://huggingface.co/blog/optimum-inference) with RoBERTa for question answering.
- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**
- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).


key: _node_content
string_value:
{"id_": "e5089f9d-e331-49f2-a178-0f7ea5baa6f2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f19435bd-da59-4db2-a96e-d489189cb136", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "5600d5c6dca955efe139ace218bf61e3b2034c83533b5923f69810afa2660ed1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0bebc9b3-9b7e-4769-8345-ead090e702b8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roberta", "title": "RoBERTa", "tokens": 3026, "retrieve_doc": true, "source": "transformers"}, "hash": "d170bbc4843d16505b2b2d462c8b143ddb0e41eff1604afe9b61b88f199bf288", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7641, "end_char_idx": 11330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`RobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**
- [`RobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

## RobertaConfig

[[autodoc]] RobertaConfig

## RobertaTokenizer

[[autodoc]] RobertaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## RobertaTokenizerFast

[[autodoc]] RobertaTokenizerFast
    - build_inputs_with_special_tokens

<frameworkcontent>
<pt>

## RobertaModel

[[autodoc]] RobertaModel
    - forward

## RobertaForCausalLM

[[autodoc]] RobertaForCausalLM
    - forward

## RobertaForMaskedLM

[[autodoc]] RobertaForMaskedLM
    - forward

## RobertaForSequenceClassification

[[autodoc]] RobertaForSequenceClassification
    - forward

## RobertaForMultipleChoice

[[autodoc]] RobertaForMultipleChoice
    - forward

## RobertaForTokenClassification

[[autodoc]] RobertaForTokenClassification
    - forward

## RobertaForQuestionAnswering

[[autodoc]] RobertaForQuestionAnswering
    - forward

</pt>
<tf>

## TFRobertaModel

[[autodoc]] TFRobertaModel
    - call

## TFRobertaForCausalLM

[[autodoc]] TFRobertaForCausalLM
    - call

## TFRobertaForMaskedLM

[[autodoc]] TFRobertaForMaskedLM
    - call

## TFRobertaForSequenceClassification

[[autodoc]] TFRobertaForSequenceClassification
    - call

## TFRobertaForMultipleChoice

[[autodoc]] TFRobertaForMultipleChoice
    - call

## TFRobertaForTokenClassification

[[autodoc]] TFRobertaForTokenClassification
    - call

## TFRobertaForQuestionAnswering

[[autodoc]] TFRobertaForQuestionAnswering
    - call

</tf>
<jax>

## FlaxRobertaModel

[[autodoc]] FlaxRobertaModel
    - __call__

## FlaxRobertaForCausalLM

[[autodoc]] FlaxRobertaForCausalLM
    - __call__

## FlaxRobertaForMaskedLM

[[autodoc]] FlaxRobertaForMaskedLM
    - __call__

## FlaxRobertaForSequenceClassification

[[autodoc]] FlaxRobertaForSequenceClassification
    - __call__

## FlaxRobertaForMultipleChoice

[[autodoc]] FlaxRobertaForMultipleChoice
    - __call__

## FlaxRobertaForTokenClassification

[[autodoc]] FlaxRobertaForTokenClassification
    - __call__

## FlaxRobertaForQuestionAnswering

[[autodoc]] FlaxRobertaForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "071dcae3-e255-4f85-b187-050686447244", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/swinv2", "title": "Swin Transformer V2", "tokens": 894, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f6ed90e-b24c-4fa1-ba43-bf7a432832f9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/swinv2", "title": "Swin Transformer V2", "tokens": 894, "retrieve_doc": true, "source": "transformers"}, "hash": "6cdd98d6b1a403b4f727ab32240655dd8b9094f2947e7a6d4566035fd54ffb19", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3253, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Swin Transformer V2

## Overview

The Swin Transformer V2 model was proposed in [Swin Transformer V2: Scaling Up Capacity and Resolution](https://arxiv.org/abs/2111.09883) by Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo.

The abstract from the paper is the following:

*Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536×1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time.*

This model was contributed by [nandwalritik](https://huggingface.co/nandwalritik).
The original code can be found [here](https://github.com/microsoft/Swin-Transformer).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Swin Transformer v2.

<PipelineTag pipeline="image-classification"/>

- [`Swinv2ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

Besides that:

- [`Swinv2ForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## Swinv2Config

[[autodoc]] Swinv2Config

## Swinv2Model

[[autodoc]] Swinv2Model
    - forward

## Swinv2ForMaskedImageModeling

[[autodoc]] Swinv2ForMaskedImageModeling
    - forward

## Swinv2ForImageClassification

[[autodoc]] transformers.Swinv2ForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "23fc9a2a-8621-43eb-92a3-745d8cfbf8bb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "hash": "6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9de0aaa6-430d-47c9-8300-0e7fa1d5f6d2", "node_type": "1", "metadata": {}, "hash": "4f2a5129568f2763e46b6b15dad6e397bc22d863f8988e1eacbfaf6ce921818b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LED

## Overview

The LED model was proposed in [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz
Beltagy, Matthew E. Peters, Arman Cohan.

The abstract from the paper is the following:

*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales
quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention
mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or
longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local
windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we
evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In
contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on
WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting
long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization
dataset.*

## Usage tips

- [`LEDForConditionalGeneration`] is an extension of
  [`BartForConditionalGeneration`] exchanging the traditional *self-attention* layer with
  *Longformer*'s *chunked self-attention* layer. [`LEDTokenizer`] is an alias of
  [`BartTokenizer`].
- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of
  1024 tokens.
- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is
  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.
- LED makes use of *global attention* by means of the `global_attention_mask` (see
  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first
  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.
- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)
  errors. This can be done by executing `model.gradient_checkpointing_enable()`. 
 Moreover, the `use_cache=False`
  flag can be used to disable the caching mechanism to save memory.
- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

## Resources

- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).
- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).


key: _node_content
string_value:
{"id_": "9de0aaa6-430d-47c9-8300-0e7fa1d5f6d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7309d6d-bb78-4fbe-a1e7-6804847f4125", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "hash": "6f16b660fabbfc08e155795602a8ab7f5609b196ea8bf201a6a4026553c0d7c3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23fc9a2a-8621-43eb-92a3-745d8cfbf8bb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/led", "title": "LED", "tokens": 1332, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f4db522a042a98ba9f0700d1e9f7c98553a052f1a26f5be81c0ec6fa890f22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1597, "end_char_idx": 4619, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`LEDTokenizer`] is an alias of
  [`BartTokenizer`].
- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of
  1024 tokens.
- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is
  gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.
- LED makes use of *global attention* by means of the `global_attention_mask` (see
  [`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first
  `<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.
- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)
  errors. This can be done by executing `model.gradient_checkpointing_enable()`. 
 Moreover, the `use_cache=False`
  flag can be used to disable the caching mechanism to save memory.
- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

## Resources

- [A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).
- [A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).
- [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## LEDConfig

[[autodoc]] LEDConfig

## LEDTokenizer

[[autodoc]] LEDTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## LEDTokenizerFast

[[autodoc]] LEDTokenizerFast

## LED specific outputs

[[autodoc]] models.led.modeling_led.LEDEncoderBaseModelOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqModelOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqLMOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqSequenceClassifierOutput

[[autodoc]] models.led.modeling_led.LEDSeq2SeqQuestionAnsweringModelOutput

[[autodoc]] models.led.modeling_tf_led.TFLEDEncoderBaseModelOutput

[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqModelOutput

[[autodoc]] models.led.modeling_tf_led.TFLEDSeq2SeqLMOutput

<frameworkcontent>
<pt>

## LEDModel

[[autodoc]] LEDModel
    - forward

## LEDForConditionalGeneration

[[autodoc]] LEDForConditionalGeneration
    - forward

## LEDForSequenceClassification

[[autodoc]] LEDForSequenceClassification
    - forward

## LEDForQuestionAnswering

[[autodoc]] LEDForQuestionAnswering
    - forward

</pt>
<tf>

## TFLEDModel

[[autodoc]] TFLEDModel
    - call

## TFLEDForConditionalGeneration

[[autodoc]] TFLEDForConditionalGeneration
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "b0e9b55f-fcf4-4eda-9b08-5d5f90c47b7a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "hash": "bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2575cd6-1e63-4fcb-bd0e-0741d807d612", "node_type": "1", "metadata": {}, "hash": "3f0fb0bf75e05b5df4469bf46b388c65e2e7dd434605d83756abf6a6b9c48d61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3023, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Phi-3

## Overview

The Phi-3 model was proposed in [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://arxiv.org/abs/2404.14219) by Microsoft.

### Summary

The abstract from the Phi-3 paper is the following:

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. The innovation lies entirely in our dataset for training, a scaled-up version of the one used for phi-2, composed of heavily filtered web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide some initial parameter-scaling results with a 7B and 14B models trained for 4.8T tokens, called phi-3-small and phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on MT-bench).

The original code for Phi-3 can be found [here](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct).

## Usage tips

- This model is very similar to `Llama` with the main difference of [`Phi3SuScaledRotaryEmbedding`] and [`Phi3YarnScaledRotaryEmbedding`], where they are used to extend the context of the rotary embeddings. The query, key and values are fused, and the MLP's up and gate projection layers are also fused.
- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.

## How to use Phi-3

<Tip warning={true}>

Phi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:

* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.

* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.

</Tip>

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

>>> messages = [{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}]
>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

>>> outputs = model.generate(inputs, max_new_tokens=32)
>>> text = tokenizer.batch_decode(outputs)[0]
>>> print(text)
<s><|user|> 
Can you provide ways to eat combinations of bananas and dragonfruits?<|end|> 
<|assistant|> 
Certainly! Bananas and dragonfruits can be combined in various delicious ways.


key: _node_content
string_value:
{"id_": "e2575cd6-1e63-4fcb-bd0e-0741d807d612", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b08a2d86-fccb-4ae5-ba8c-a2b4bcb0613a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "hash": "bd10ad10af0e8b050bfea8d3953598adc79825f145963f32a162c4946178a2bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0e9b55f-fcf4-4eda-9b08-5d5f90c47b7a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi3", "title": "Phi-3", "tokens": 1045, "retrieve_doc": true, "source": "transformers"}, "hash": "23f8da943fcf1bf639bae95c6dd74a7f324beac8f61e3b19049bb70c83c8913f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1434, "end_char_idx": 3482, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The query, key and values are fused, and the MLP's up and gate projection layers are also fused.
- The tokenizer used for this model is identical to the [`LlamaTokenizer`], with the exception of additional tokens.

## How to use Phi-3

<Tip warning={true}>

Phi-3 has been integrated in the development version (4.40.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:

* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.

* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.

</Tip>

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

>>> messages = [{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}]
>>> inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

>>> outputs = model.generate(inputs, max_new_tokens=32)
>>> text = tokenizer.batch_decode(outputs)[0]
>>> print(text)
<s><|user|> 
Can you provide ways to eat combinations of bananas and dragonfruits?<|end|> 
<|assistant|> 
Certainly! Bananas and dragonfruits can be combined in various delicious ways. Here are some ideas for eating combinations of bananas and
```

## Phi3Config

[[autodoc]] Phi3Config

<frameworkcontent>
<pt>

## Phi3Model

[[autodoc]] Phi3Model
    - forward

## Phi3ForCausalLM

[[autodoc]] Phi3ForCausalLM
    - forward
    - generate

## Phi3ForSequenceClassification

[[autodoc]] Phi3ForSequenceClassification
    - forward

## Phi3ForTokenClassification

[[autodoc]] Phi3ForTokenClassification
    - forward

</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "cd357d90-8776-463e-aceb-140aaf0cce2b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26da89fb-5541-451c-9a26-1416c49d435b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d2356d8-b2bd-47f0-aa3f-5699452b16c6", "node_type": "1", "metadata": {}, "hash": "3468b4d14c4f66f7d2e223f78298ebb162aeca0c46267819de0f7399ae3dcf4d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3143, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# UPerNet

## Overview

The UPerNet model was proposed in [Unified Perceptual Parsing for Scene Understanding](https://arxiv.org/abs/1807.10221)
by Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun. UPerNet is a general framework to effectively segment
a wide range of concepts from images, leveraging any vision backbone like [ConvNeXt](convnext) or [Swin](swin).

The abstract from the paper is the following:

*Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/upernet_architecture.jpg"
alt="drawing" width="600"/>

<small> UPerNet framework. Taken from the <a href="https://arxiv.org/abs/1807.10221">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).

## Usage examples

UPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:

```py
from transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = SwinConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

To use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:

```py
from transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = ConvNextConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

Note that this will randomly initialize all the weights of the model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with UPerNet.

- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).
- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb).


key: _node_content
string_value:
{"id_": "6d2356d8-b2bd-47f0-aa3f-5699452b16c6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26da89fb-5541-451c-9a26-1416c49d435b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "cd643830d234287828240de9e7b16ccfb20df55bcc067de7290ce5dac9ba6b05", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cd357d90-8776-463e-aceb-140aaf0cce2b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/upernet", "title": "UPerNet", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "38138b6f220a5f7ad2bf742f55a19813bb1c080fdc9850e3b7342f260765aa71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1542, "end_char_idx": 3588, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code is based on OpenMMLab's mmsegmentation [here](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/models/decode_heads/uper_head.py).

## Usage examples

UPerNet is a general framework for semantic segmentation. It can be used with any vision backbone, like so:

```py
from transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = SwinConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

To use another vision backbone, like [ConvNeXt](convnext), simply instantiate the model with the appropriate backbone:

```py
from transformers import ConvNextConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = ConvNextConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

Note that this will randomly initialize all the weights of the model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with UPerNet.

- Demo notebooks for UPerNet can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UPerNet).
- [`UperNetForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/semantic_segmentation.ipynb).
- See also: [Semantic segmentation task guide](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## UperNetConfig

[[autodoc]] UperNetConfig

## UperNetForSemanticSegmentation

[[autodoc]] UperNetForSemanticSegmentation
    - forward


key: _node_content
string_value:
{"id_": "12db5492-1028-4127-a735-a139a063c096", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "hash": "36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0f1e826-8bcf-47b4-9b4f-cf5ef9f00a4b", "node_type": "1", "metadata": {}, "hash": "1671e10b84b6b2ba565fc0072b3d01791f6449488c59a86f81a1d80eef670c47", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Blenderbot

## Overview

The Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,
Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.

The abstract of the paper is the following:

*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that
scaling neural models in the number of parameters and the size of the data they are trained on gives improved results,
we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of
skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to
their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent
persona. We show that large scale models can learn these skills when given appropriate training data and choice of
generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models
and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn
dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing
failure cases of our models.*

This model was contributed by [sshleifer](https://huggingface.co/sshleifer). The authors' code can be found [here](https://github.com/facebookresearch/ParlAI) .

## Usage tips and example

Blenderbot is a model with absolute position embeddings so it's usually advised to pad the inputs on the right 
rather than the left.

An example:

```python
>>> from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration

>>> mname = "facebook/blenderbot-400M-distill"
>>> model = BlenderbotForConditionalGeneration.from_pretrained(mname)
>>> tokenizer = BlenderbotTokenizer.from_pretrained(mname)
>>> UTTERANCE = "My friends are cool but they eat too many carbs."
>>> inputs = tokenizer([UTTERANCE], return_tensors="pt")
>>> reply_ids = model.generate(**inputs)
>>> print(tokenizer.batch_decode(reply_ids))
["<s> That's unfortunate. Are they trying to lose weight or are they just trying to be healthier?</s>"]
```

## Implementation Notes

- Blenderbot uses a standard [seq2seq model transformer](https://arxiv.org/pdf/1706.03762.pdf) based architecture.
- Available checkpoints can be found in the [model hub](https://huggingface.co/models?search=blenderbot).
- This is the *default* Blenderbot model class. However, some smaller checkpoints, such as
  `facebook/blenderbot_small_90M`, have a different architecture and consequently should be used with
  [BlenderbotSmall](blenderbot-small).

  
## Resources

- [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## BlenderbotConfig

[[autodoc]] BlenderbotConfig

## BlenderbotTokenizer

[[autodoc]] BlenderbotTokenizer
    - build_inputs_with_special_tokens

## BlenderbotTokenizerFast

[[autodoc]] BlenderbotTokenizerFast
    - build_inputs_with_special_tokens


key: _node_content
string_value:
{"id_": "c0f1e826-8bcf-47b4-9b4f-cf5ef9f00a4b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ca49c4f-2548-485b-a3b0-1b1c33ede8ec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "hash": "36b1057c151a6bc23415ea0a42657e3f6026eb69a61e5c726366c6a125055b90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12db5492-1028-4127-a735-a139a063c096", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot", "title": "Blenderbot", "tokens": 1155, "retrieve_doc": true, "source": "transformers"}, "hash": "086afa922d1464c5c9f1c1a4180fe3020cf6b8ead67cd6a88a0969880d5d6cff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3297, "end_char_idx": 4181, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>

## BlenderbotModel

See [`~transformers.BartModel`] for arguments to *forward* and *generate*

[[autodoc]] BlenderbotModel
    - forward

## BlenderbotForConditionalGeneration

See [`~transformers.BartForConditionalGeneration`] for arguments to *forward* and *generate*

[[autodoc]] BlenderbotForConditionalGeneration
    - forward

## BlenderbotForCausalLM

[[autodoc]] BlenderbotForCausalLM
    - forward

</pt>
<tf>

## TFBlenderbotModel

[[autodoc]] TFBlenderbotModel
    - call

## TFBlenderbotForConditionalGeneration

[[autodoc]] TFBlenderbotForConditionalGeneration
    - call

</tf>
<jax>

## FlaxBlenderbotModel

[[autodoc]] FlaxBlenderbotModel
    - __call__
    - encode
    - decode

## FlaxBlenderbotForConditionalGeneration

[[autodoc]] FlaxBlenderbotForConditionalGeneration
    - __call__
    - encode
    - decode

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "c9283ffe-deae-493d-a0f0-5a2ad15ab62f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "hash": "c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eb65f290-46d1-4c2b-afb2-fc92da7f27f1", "node_type": "1", "metadata": {}, "hash": "7c86b7e34c683615d4aa424eaea5958ee97a6b3d0998581f868d7d53f1585a1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2871, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Splinter

## Overview

The Splinter model was proposed in [Few-Shot Question Answering by Pretraining Span Selection](https://arxiv.org/abs/2101.00438) by Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy. Splinter
is an encoder-only transformer (similar to BERT) pretrained using the recurring span selection task on a large corpus
comprising Wikipedia and the Toronto Book Corpus.

The abstract from the paper is the following:

In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order
of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred
training examples are available, and observe that standard models perform poorly, highlighting the discrepancy between
current pretraining objectives and question answering. We propose a new pretraining scheme tailored for question
answering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all
recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans
are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select
the answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD
with only 128 training examples), while maintaining competitive performance in the high-resource setting.

This model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).

## Usage tips

- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize
  to question representations which are used to predict the answers. This layer is called QASS, and is the default
  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:
- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already
  contains this special token. Also, its default behavior is to use this token when two sequences are given (for
  example, in the *run_qa.py* script).
- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for
  the success of your model, especially in a few-shot setting.
- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that
  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one
  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at
  fine-tuning, as it is shown to yield better results for some cases in the paper.


key: _node_content
string_value:
{"id_": "eb65f290-46d1-4c2b-afb2-fc92da7f27f1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "46ba8514-f602-4b2f-b0cd-0830de8369e2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "hash": "c62477bd3a8359d7de5736bcc1c7e4bc94f775cd16f999207061da2a55290933", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c9283ffe-deae-493d-a0f0-5a2ad15ab62f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/splinter", "title": "Splinter", "tokens": 961, "retrieve_doc": true, "source": "transformers"}, "hash": "a51b64a63b5822aa0dd685583af29628804befb10d84db6ab5442ab5a36d7068", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1291, "end_char_idx": 3469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD
with only 128 training examples), while maintaining competitive performance in the high-resource setting.

This model was contributed by [yuvalkirstain](https://huggingface.co/yuvalkirstain) and [oriram](https://huggingface.co/oriram). The original code can be found [here](https://github.com/oriram/splinter).

## Usage tips

- Splinter was trained to predict answers spans conditioned on a special [QUESTION] token. These tokens contextualize
  to question representations which are used to predict the answers. This layer is called QASS, and is the default
  behaviour in the [`SplinterForQuestionAnswering`] class. Therefore:
- Use [`SplinterTokenizer`] (rather than [`BertTokenizer`]), as it already
  contains this special token. Also, its default behavior is to use this token when two sequences are given (for
  example, in the *run_qa.py* script).
- If you plan on using Splinter outside *run_qa.py*, please keep in mind the question token - it might be important for
  the success of your model, especially in a few-shot setting.
- Please note there are two different checkpoints for each size of Splinter. Both are basically the same, except that
  one also has the pretrained weights of the QASS layer (*tau/splinter-base-qass* and *tau/splinter-large-qass*) and one
  doesn't (*tau/splinter-base* and *tau/splinter-large*). This is done to support randomly initializing this layer at
  fine-tuning, as it is shown to yield better results for some cases in the paper.

## Resources

- [Question answering task guide](../tasks/question-answering)

## SplinterConfig

[[autodoc]] SplinterConfig

## SplinterTokenizer

[[autodoc]] SplinterTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## SplinterTokenizerFast

[[autodoc]] SplinterTokenizerFast

## SplinterModel

[[autodoc]] SplinterModel
    - forward

## SplinterForQuestionAnswering

[[autodoc]] SplinterForQuestionAnswering
    - forward

## SplinterForPreTraining

[[autodoc]] SplinterForPreTraining
    - forward


key: _node_content
string_value:
{"id_": "537189bf-44a5-452b-8e84-dbfe66c5da74", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sew-d", "title": "SEW-D", "tokens": 637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b680134e-3f48-4dac-9dda-1d25c20ecf6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/sew-d", "title": "SEW-D", "tokens": 637, "retrieve_doc": true, "source": "transformers"}, "hash": "7932e84258bad487dcc35ad86f38719dc0750fa3f4768f89de9adc2e5acbcf9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SEW-D

## Overview

SEW-D (Squeezed and Efficient Wav2Vec with Disentangled attention) was proposed in [Performance-Efficiency Trade-offs
in Unsupervised Pre-training for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim,
Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.

The abstract from the paper is the following:

*This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition
(ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance
and its efficiency. Putting together all our observations, we introduce SEW (Squeezed and Efficient Wav2vec), a
pre-trained model architecture with significant improvements along both performance and efficiency dimensions across a
variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW achieves a 1.9x
inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference
time, SEW reduces word error rate by 25-50% across different model sizes.*

This model was contributed by [anton-l](https://huggingface.co/anton-l).

## Usage tips

- SEW-D is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- SEWDForCTC is fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].

## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## SEWDConfig

[[autodoc]] SEWDConfig

## SEWDModel

[[autodoc]] SEWDModel
    - forward

## SEWDForCTC

[[autodoc]] SEWDForCTC
    - forward

## SEWDForSequenceClassification

[[autodoc]] SEWDForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "6f7cc510-098e-4c53-8aa3-835452211709", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b938a64-4c32-45fb-bc52-8bb8716f8d1c", "node_type": "1", "metadata": {}, "hash": "b8e74e9d13d72f4bf23603a5fc8ef2560b7c5797615084d281d7ce1ee71f0eb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TAPAS

## Overview

The TAPAS model was proposed in [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://www.aclweb.org/anthology/2020.acl-main.398)
by Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and Julian Martin Eisenschlos. It's a BERT-based model specifically 
designed (and pre-trained) for answering questions about tabular data. Compared to BERT, TAPAS uses relative position embeddings and has 7 
token types that encode tabular structure. TAPAS is pre-trained on the masked language modeling (MLM) objective on a large dataset comprising 
millions of tables from English Wikipedia and corresponding texts. 

For question answering, TAPAS has 2 heads on top: a cell selection head and an aggregation head, for (optionally) performing aggregations (such as counting or summing) among selected cells. TAPAS has been fine-tuned on several datasets: 
- [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) (Sequential Question Answering by Microsoft)
- [WTQ](https://github.com/ppasupat/WikiTableQuestions) (Wiki Table Questions by Stanford University)
- [WikiSQL](https://github.com/salesforce/WikiSQL) (by Salesforce). 

It achieves state-of-the-art on both SQA and WTQ, while having comparable performance to SOTA on WikiSQL, with a much simpler architecture.

The abstract from the paper is the following:

*Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*

In addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task).


key: _node_content
string_value:
{"id_": "4b938a64-4c32-45fb-bc52-8bb8716f8d1c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f7cc510-098e-4c53-8aa3-835452211709", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "d6b3cdf8449415b8bbe646328e23bedbdef5b9b067de10e76e87a9d3965719f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30324346-5201-4150-b427-9e6b43cbad6e", "node_type": "1", "metadata": {}, "hash": "54f75347f167662c9e9b12b575e19602357874423cd0452b5e98e5f7693efd94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1628, "end_char_idx": 5153, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIKISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.*

In addition, the authors have further pre-trained TAPAS to recognize **table entailment**, by creating a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. The authors of TAPAS call this further pre-training intermediate pre-training (since TAPAS is first pre-trained on MLM, and then on another dataset). They found that intermediate pre-training further improves performance on SQA, achieving a new state-of-the-art as well as state-of-the-art on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking), a large-scale dataset with 16k Wikipedia tables for table entailment (a binary classification task). For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png"
alt="drawing" width="600"/> 

<small> TAPAS architecture. Taken from the <a href="https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html">original blog post</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).

## Usage tips

- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision="no_reset"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.


key: _node_content
string_value:
{"id_": "30324346-5201-4150-b427-9e6b43cbad6e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b938a64-4c32-45fb-bc52-8bb8716f8d1c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "c5c67b023e5d77d4ef4825f5cda00f03a40a730ffba7398b1d44095406639359", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f", "node_type": "1", "metadata": {}, "hash": "9e7e7fff44098dce7ab89839730c10a6d7350eb0dbf2808b4123757f6337765e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3458, "end_char_idx": 6633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For more details, see their follow-up paper: [Understanding tables with intermediate pre-training](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) by Julian Martin Eisenschlos, Syrine Krichene and Thomas Müller.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tapas_architecture.png"
alt="drawing" width="600"/> 

<small> TAPAS architecture. Taken from the <a href="https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html">original blog post</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The Tensorflow version of this model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/tapas).

## Usage tips

- TAPAS is a model that uses relative position embeddings by default (restarting the position embeddings at every cell of the table). Note that this is something that was added after the publication of the original TAPAS paper. According to the authors, this usually results in a slightly better performance, and allows you to encode longer sequences without running out of embeddings. This is reflected in the `reset_position_index_per_cell` parameter of [`TapasConfig`], which is set to `True` by default. The default versions of the models available on the [hub](https://huggingface.co/models?search=tapas) all use relative position embeddings. You can still use the ones with absolute position embeddings by passing in an additional argument `revision="no_reset"` when calling the `from_pretrained()` method. Note that it's usually advised to pad the inputs on the right rather than the left.
- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).
- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as "what is his age?" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See "Usage" section for more info.
- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.

## Usage: fine-tuning

Here we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.


key: _node_content
string_value:
{"id_": "ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30324346-5201-4150-b427-9e6b43cbad6e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "53b4fd07ba325d6af22ef1bdf229cf88a811f6466c00b0f19f5e20000ad0dc14", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2bc5b3c4-bb1b-48b6-be34-6c4f0de404af", "node_type": "1", "metadata": {}, "hash": "5853700e59e5aaa95a01af1e6d806a23720f14a9e488a4c321261854b4657eb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5070, "end_char_idx": 8015, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that it's usually advised to pad the inputs on the right rather than the left.
- TAPAS is based on BERT, so `TAPAS-base` for example corresponds to a `BERT-base` architecture. Of course, `TAPAS-large` will result in the best performance (the results reported in the paper are from `TAPAS-large`). Results of the various sized models are shown on the [original GitHub repository](https://github.com/google-research/tapas).
- TAPAS has checkpoints fine-tuned on SQA, which are capable of answering questions related to a table in a conversational set-up. This means that you can ask follow-up questions such as "what is his age?" related to the previous question. Note that the forward pass of TAPAS is a bit different in case of a conversational set-up: in that case, you have to feed every table-question pair one by one to the model, such that the `prev_labels` token type ids can be overwritten by the predicted `labels` of the model to the previous question. See "Usage" section for more info.
- TAPAS is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.

## Usage: fine-tuning

Here we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.

**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**

Basically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:

1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask "what's the name of the first actor?" then you can ask a follow-up question such as "how old is he?". Here, questions do not involve any aggregation (all questions are cell selection questions).
2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask "what's the total number of goals Cristiano Ronaldo made in his career?". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.
3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.


key: _node_content
string_value:
{"id_": "2bc5b3c4-bb1b-48b6-be34-6c4f0de404af", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba8dc3d0-8e79-4ad9-8100-6577a1c6a30f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "815cca00555d4e49fb99d6f06cd5627dfd7d9a9b09a739e7cefab84e6b0e10d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b", "node_type": "1", "metadata": {}, "hash": "07c2ee055ce6f362a7410769e0cd6fd2cd1852b46c1f2867f2f2e1141acb5fcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6286, "end_char_idx": 10123, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Models trained with a causal language modeling (CLM) objective are better in that regard. Note that TAPAS can be used as an encoder in the EncoderDecoderModel framework, to combine it with an autoregressive text decoder such as GPT-2.

## Usage: fine-tuning

Here we explain how you can fine-tune [`TapasForQuestionAnswering`] on your own dataset.

**STEP 1: Choose one of the 3 ways in which you can use TAPAS - or experiment**

Basically, there are 3 different ways in which one can fine-tune [`TapasForQuestionAnswering`], corresponding to the different datasets on which Tapas was fine-tuned:

1. SQA: if you're interested in asking follow-up questions related to a table, in a conversational set-up. For example if you first ask "what's the name of the first actor?" then you can ask a follow-up question such as "how old is he?". Here, questions do not involve any aggregation (all questions are cell selection questions).
2. WTQ: if you're not interested in asking questions in a conversational set-up, but rather just asking questions related to a table, which might involve aggregation, such as counting a number of rows, summing up cell values or averaging cell values. You can then for example ask "what's the total number of goals Cristiano Ronaldo made in his career?". This case is also called **weak supervision**, since the model itself must learn the appropriate aggregation operator (SUM/COUNT/AVERAGE/NONE) given only the answer to the question as supervision.
3. WikiSQL-supervised: this dataset is based on WikiSQL with the model being given the ground truth aggregation operator during training. This is also called **strong supervision**. Here, learning the appropriate aggregation operator is much easier.

To summarize:

| **Task**                            | **Example dataset** | **Description**                                                                                         |
|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|
| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |
| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |

<frameworkcontent>
<pt>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way.


key: _node_content
string_value:
{"id_": "b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2bc5b3c4-bb1b-48b6-be34-6c4f0de404af", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "f4f4f0d59938eaf215ec69c506e83cc666ba1a7ab549265cf9d18100f1754588", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f32b111-f057-4d11-8086-c89cd0efbe1b", "node_type": "1", "metadata": {}, "hash": "6bae031756ce043224ba32faee981850fd67269f509027fa69a07b2471954c4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8017, "end_char_idx": 11784, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To summarize:

| **Task**                            | **Example dataset** | **Description**                                                                                         |
|-------------------------------------|---------------------|---------------------------------------------------------------------------------------------------------|
| Conversational                      | SQA                 | Conversational, only cell selection questions                                                           |
| Weak supervision for aggregation    | WTQ                 | Questions might involve aggregation, and the model must learn this given only the answer as supervision |
| Strong supervision for aggregation  | WikiSQL-supervised  | Questions might involve aggregation, and the model must learn this given the gold aggregation operator  |

<frameworkcontent>
<pt>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below.

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</pt>
<tf>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration.


key: _node_content
string_value:
{"id_": "4f32b111-f057-4d11-8086-c89cd0efbe1b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b41bf89b-8e27-48bc-8ad6-f8b5e11b1e6b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "a2f98c5841119cccaec3faf68f8769ed6e19592332ca63edf45023e853fcdcd0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82c2d1ef-66f7-42c5-937e-f3f8cedd4f52", "node_type": "1", "metadata": {}, "hash": "729ddea90a9e793940fcc8f071cd5c8cec97e6b6c3d3641c1020104c276daeeb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10124, "end_char_idx": 13467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here's an example:

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</pt>
<tf>
Initializing a model with a pre-trained base and randomly initialized classification heads from the hub can be done as shown below. Be sure to have installed the [tensorflow_probability](https://github.com/tensorflow/probability) dependency:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # for example, the base sized model with default SQA configuration
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base")

>>> # or, the base sized model with WTQ configuration
>>> config = TapasConfig.from_pretrained("google/tapas-base-finetuned-wtq")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> # or, the base sized model with WikiSQL configuration
>>> config = TapasConfig("google-base-finetuned-wikisql-supervised")
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```

Of course, you don't necessarily have to follow one of these three ways in which TAPAS was fine-tuned. You can also experiment by defining any hyperparameters you want when initializing [`TapasConfig`], and then create a [`TFTapasForQuestionAnswering`] based on that configuration. For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</tf>
</frameworkcontent>

What you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.

For a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).

**STEP 2: Prepare your data in the SQA format**

Second, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:

- `id`: optional, id of the table-question pair, for bookkeeping purposes.
- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA).


key: _node_content
string_value:
{"id_": "82c2d1ef-66f7-42c5-937e-f3f8cedd4f52", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f32b111-f057-4d11-8086-c89cd0efbe1b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "27ff86a8e9e17bf77ea19de0b8d828369a12cf2267cb19533f4965482377d8c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f82c876-0908-4433-bc7c-cf61bf31a99e", "node_type": "1", "metadata": {}, "hash": "05338a496a08ff547ae8a53b5860be98e6963b991e8c251e144094cd293cd2fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11785, "end_char_idx": 14917, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have a dataset that has both conversational questions and questions that might involve aggregation, then you can do it this way. Here's an example:

```py
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # you can initialize the classification heads any way you want (see docs of TapasConfig)
>>> config = TapasConfig(num_aggregation_labels=3, average_logits_per_cell=True)
>>> # initializing the pre-trained base sized model with our custom classification heads
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)
```
</tf>
</frameworkcontent>

What you can also do is start from an already fine-tuned checkpoint. A note here is that the already fine-tuned checkpoint on WTQ has some issues due to the L2-loss which is somewhat brittle. See [here](https://github.com/google-research/tapas/issues/91#issuecomment-735719340) for more info.

For a list of all pre-trained and fine-tuned TAPAS checkpoints available on HuggingFace's  hub, see [here](https://huggingface.co/models?search=tapas).

**STEP 2: Prepare your data in the SQA format**

Second, no matter what you picked above, you should prepare your dataset in the [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253) format. This format is a TSV/CSV file with the following columns:

- `id`: optional, id of the table-question pair, for bookkeeping purposes.
- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.
- `question`: string
- `table_file`: string, name of a csv file containing the tabular data
- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)
- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)
- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)
- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)

The tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.


key: _node_content
string_value:
{"id_": "3f82c876-0908-4433-bc7c-cf61bf31a99e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82c2d1ef-66f7-42c5-937e-f3f8cedd4f52", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "ef8ce7b6fc8cd8e36015416868943c87a105d5d92367cff769a952cc9e61d992", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "789f1f8f-f587-4c44-a7d0-4d885059bdc6", "node_type": "1", "metadata": {}, "hash": "2ad94ff4bcb398e41591ae5d15388101156198403251862b34b3b4bf1f7e1630", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13205, "end_char_idx": 16801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `annotator`: optional, id of the person who annotated the table-question pair, for bookkeeping purposes.
- `position`: integer indicating if the question is the first, second, third,... related to the table. Only required in case of conversational setup (SQA). You don't need this column in case you're going for WTQ/WikiSQL-supervised.
- `question`: string
- `table_file`: string, name of a csv file containing the tabular data
- `answer_coordinates`: list of one or more tuples (each tuple being a cell coordinate, i.e. row, column pair that is part of the answer)
- `answer_text`: list of one or more strings (each string being a cell value that is part of the answer)
- `aggregation_label`: index of the aggregation operator. Only required in case of strong supervision for aggregation (the WikiSQL-supervised case)
- `float_answer`: the float answer to the question, if there is one (np.nan if there isn't). Only required in case of weak supervision for aggregation (such as WTQ and WikiSQL)

The tables themselves should be present in a folder, each table being a separate csv file. Note that the authors of the TAPAS algorithm used conversion scripts with some automated logic to convert the other datasets (WTQ, WikiSQL) into the SQA format. The author explains this [here](https://github.com/google-research/tapas/issues/50#issuecomment-705465960). A conversion of this script that works with HuggingFace's implementation can be found [here](https://github.com/NielsRogge/tapas_utils). Interestingly, these conversion scripts are not perfect (the `answer_coordinates` and `float_answer` fields are populated based on the `answer_text`), meaning that WTQ and WikiSQL results could actually be improved.

**STEP 3: Convert your data into tensors using TapasTokenizer**

<frameworkcontent>
<pt>
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different
inputs to be fine-tuned:

| **Task**                           | **Required inputs**                                                                                                 |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |
|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |
| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |

[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:

```py
>>> from transformers import TapasTokenizer
>>> import pandas as pd

>>> model_name = "google/tapas-base"
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?


key: _node_content
string_value:
{"id_": "789f1f8f-f587-4c44-a7d0-4d885059bdc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f82c876-0908-4433-bc7c-cf61bf31a99e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "197055e2896a9033362415f589ac2647bd715cce8e27b70f6190a9fc6a4da435", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8290ccc1-565f-4010-b928-6f9b10901094", "node_type": "1", "metadata": {}, "hash": "1aca70a288eb127f649e47744011c0fccf867966ce500164d4cab620c34e7516", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14919, "end_char_idx": 17968, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**STEP 3: Convert your data into tensors using TapasTokenizer**

<frameworkcontent>
<pt>
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TapasForQuestionAnswering`] requires different
inputs to be fine-tuned:

| **Task**                           | **Required inputs**                                                                                                 |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |
|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |
| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |

[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:

```py
>>> from transformers import TapasTokenizer
>>> import pandas as pd

>>> model_name = "google/tapas-base"
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table,
...     queries=queries,
...     answer_coordinates=answer_coordinates,
...     answer_text=answer_text,
...     padding="max_length",
...     return_tensors="pt",
... )
>>> inputs
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}
```

Note that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import torch
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


key: _node_content
string_value:
{"id_": "8290ccc1-565f-4010-b928-6f9b10901094", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "789f1f8f-f587-4c44-a7d0-4d885059bdc6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "644bc8e515abf499a9c8ca27f59b15cd173f241d13ec900295ad21382409ea96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56a9ed95-5739-4be9-8122-eb895710efb6", "node_type": "1", "metadata": {}, "hash": "10ab27f0dffc801c0708d0f5be38c1d5b4d944fce26212363e9b01a37c3921c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16801, "end_char_idx": 19106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table,
...     queries=queries,
...     answer_coordinates=answer_coordinates,
...     answer_text=answer_text,
...     padding="max_length",
...     return_tensors="pt",
... )
>>> inputs
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}
```

Note that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import torch
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


>>> class TableDataset(torch.utils.data.Dataset):
...     def __init__(self, data, tokenizer):
...         self.data = data
...         self.tokenizer = tokenizer

...     def __getitem__(self, idx):
...         item = data.iloc[idx]
...         table = pd.read_csv(table_csv_path + item.table_file).astype(
...             str
...         )  # be sure to make your table data text only
...         encoding = self.tokenizer(
...             table=table,
...             queries=item.question,
...             answer_coordinates=item.answer_coordinates,
...             answer_text=item.answer_text,
...             truncation=True,
...             padding="max_length",
...             return_tensors="pt",
...         )
...         # remove the batch dimension which the tokenizer adds by default
...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}
...         # add the float_answer which is also required (weak supervision for aggregation case)
...         encoding["float_answer"] = torch.tensor(item.float_answer)
...         return encoding

...     def __len__(self):
...         return len(self.data)


key: _node_content
string_value:
{"id_": "56a9ed95-5739-4be9-8122-eb895710efb6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8290ccc1-565f-4010-b928-6f9b10901094", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "22befb726717d656ae180ac0772117d7bf38d69df35118d15f74fd8f5aea19eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8c50a54-ee4c-474b-ae3a-b863cfa7a282", "node_type": "1", "metadata": {}, "hash": "5490a13e6d08534c3f45d0017a5a95613556f89727d1492687d4ad4cac0f95ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17600, "end_char_idx": 22271, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import torch
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


>>> class TableDataset(torch.utils.data.Dataset):
...     def __init__(self, data, tokenizer):
...         self.data = data
...         self.tokenizer = tokenizer

...     def __getitem__(self, idx):
...         item = data.iloc[idx]
...         table = pd.read_csv(table_csv_path + item.table_file).astype(
...             str
...         )  # be sure to make your table data text only
...         encoding = self.tokenizer(
...             table=table,
...             queries=item.question,
...             answer_coordinates=item.answer_coordinates,
...             answer_text=item.answer_text,
...             truncation=True,
...             padding="max_length",
...             return_tensors="pt",
...         )
...         # remove the batch dimension which the tokenizer adds by default
...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}
...         # add the float_answer which is also required (weak supervision for aggregation case)
...         encoding["float_answer"] = torch.tensor(item.float_answer)
...         return encoding

...     def __len__(self):
...         return len(self.data)


>>> data = pd.read_csv(tsv_path, sep="\t")
>>> train_dataset = TableDataset(data, tokenizer)
>>> train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)
```
</pt>
<tf>
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TFTapasForQuestionAnswering`] requires different
inputs to be fine-tuned:

| **Task**                           | **Required inputs**                                                                                                 |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |
|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |
| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |

[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:

```py
>>> from transformers import TapasTokenizer
>>> import pandas as pd

>>> model_name = "google/tapas-base"
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table,
...     queries=queries,
...     answer_coordinates=answer_coordinates,
...     answer_text=answer_text,
...     padding="max_length",
...     return_tensors="tf",
... )
>>> inputs
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}
```

Note that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import tensorflow as tf
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


key: _node_content
string_value:
{"id_": "e8c50a54-ee4c-474b-ae3a-b863cfa7a282", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56a9ed95-5739-4be9-8122-eb895710efb6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "d3bef3c03a0bb62fdfd22809525e3b40d9573bf3c045bc3fcc165f1da3ad634f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c287df5a-1d01-40d3-8134-34d4179a0f23", "node_type": "1", "metadata": {}, "hash": "bf3e72011ba82d8e9164c074330aba58095415c84d7dd5ec0f3f6aa5c8b619c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> class TableDataset:
...     def __init__(self, data, tokenizer):
...         self.data = data
...         self.tokenizer = tokenizer

...     def __iter__(self):
...         for idx in range(self.__len__()):
...             item = self.data.iloc[idx]
...             table = pd.read_csv(table_csv_path + item.table_file).astype(
...                 str
...             )  # be sure to make your table data text only
...             encoding = self.tokenizer(
...                 table=table,
...                 queries=item.question,
...                 answer_coordinates=item.answer_coordinates,
...                 answer_text=item.answer_text,
...                 truncation=True,
...                 padding="max_length",
...                 return_tensors="tf",
...             )
...             # remove the batch dimension which the tokenizer adds by default
...             encoding = {key: tf.squeeze(val, 0) for key, val in encoding.items()}
...             # add the float_answer which is also required (weak supervision for aggregation case)
...             encoding["float_answer"] = tf.convert_to_tensor(item.float_answer, dtype=tf.float32)
...             yield encoding["input_ids"], encoding["attention_mask"], encoding["numeric_values"], encoding[
...                 "numeric_values_scale"
...             ], encoding["token_type_ids"], encoding["labels"], encoding["float_answer"]

...     def __len__(self):
...         return len(self.data)


>>> data = pd.read_csv(tsv_path, sep="\t")
>>> train_dataset = TableDataset(data, tokenizer)
>>> output_signature = (
...     tf.TensorSpec(shape=(512,), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.float32),
...     tf.TensorSpec(shape=(512,), dtype=tf.float32),
...     tf.TensorSpec(shape=(512, 7), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.float32),
... )
>>> train_dataloader = tf.data.Dataset.from_generator(train_dataset, output_signature=output_signature).batch(32)
```
</tf>
</frameworkcontent>

Note that here, we encode each table-question pair independently. This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`
index) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.

**STEP 4: Train (fine-tune) the model

<frameworkcontent>
<pt>
You can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.


key: _node_content
string_value:
{"id_": "c287df5a-1d01-40d3-8134-34d4179a0f23", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8c50a54-ee4c-474b-ae3a-b863cfa7a282", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "9287cc4bb0f5f09d33e75fe4210174a0942bac5a08a0ea1aae8e3dbcd6613642", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d548484a-2102-47a0-9ba8-f41b99635cfe", "node_type": "1", "metadata": {}, "hash": "473e2b92ae58897c38b18d4a0d30fd273ca0a21b89f9be891ebcc87d865bcc4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`
index) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.

**STEP 4: Train (fine-tune) the model

<frameworkcontent>
<pt>
You can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.     init_cell_selection_weights_to_zero=True,
.     select_one_column=True,
.     allow_empty_column_selection=False,
.     temperature=0.0352513,
. )
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> optimizer = AdamW(model.parameters(), lr=5e-5)

>>> model.train()
>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch["input_ids"]
.         attention_mask = batch["attention_mask"]
.         token_type_ids = batch["token_type_ids"]
.         labels = batch["labels"]
.         numeric_values = batch["numeric_values"]
.         numeric_values_scale = batch["numeric_values_scale"]
.         float_answer = batch["float_answer"]

.         # zero the parameter gradients
.         optimizer.zero_grad()

.         # forward + backward + optimize
.         outputs = model(
.             input_ids=input_ids,
.             attention_mask=attention_mask,
.             token_type_ids=token_type_ids,
.             labels=labels,
.             numeric_values=numeric_values,
.             numeric_values_scale=numeric_values_scale,
.             float_answer=float_answer,
.         )
.         loss = outputs.loss
.         loss.backward()
.         optimizer.step()
```
</pt>
<tf>
You can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> import tensorflow as tf
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.


key: _node_content
string_value:
{"id_": "d548484a-2102-47a0-9ba8-f41b99635cfe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c287df5a-1d01-40d3-8134-34d4179a0f23", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "79d4a4602f87a5f2395579edb72aaf4fc3b8c112d896e5a6c9d27da941f0874c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8449d6d-408d-4d6c-8dab-85cbb0005a65", "node_type": "1", "metadata": {}, "hash": "13daf7eb36d0d077b78ee3959811fe83697d4b661e94c8eca429d1b6b7bb72f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
from_pretrained("google/tapas-base", config=config)

>>> optimizer = AdamW(model.parameters(), lr=5e-5)

>>> model.train()
>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch["input_ids"]
.         attention_mask = batch["attention_mask"]
.         token_type_ids = batch["token_type_ids"]
.         labels = batch["labels"]
.         numeric_values = batch["numeric_values"]
.         numeric_values_scale = batch["numeric_values_scale"]
.         float_answer = batch["float_answer"]

.         # zero the parameter gradients
.         optimizer.zero_grad()

.         # forward + backward + optimize
.         outputs = model(
.             input_ids=input_ids,
.             attention_mask=attention_mask,
.             token_type_ids=token_type_ids,
.             labels=labels,
.             numeric_values=numeric_values,
.             numeric_values_scale=numeric_values_scale,
.             float_answer=float_answer,
.         )
.         loss = outputs.loss
.         loss.backward()
.         optimizer.step()
```
</pt>
<tf>
You can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> import tensorflow as tf
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.     init_cell_selection_weights_to_zero=True,
.     select_one_column=True,
.     allow_empty_column_selection=False,
.     temperature=0.0352513,
. )
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)

>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch[0]
.         attention_mask = batch[1]
.         token_type_ids = batch[4]
.         labels = batch[-1]
.         numeric_values = batch[2]
.         numeric_values_scale = batch[3]
.         float_answer = batch[6]

.         # forward + backward + optimize
.         with tf.GradientTape() as tape:
.             outputs = model(
.                 input_ids=input_ids,
.                 attention_mask=attention_mask,
.                 token_type_ids=token_type_ids,
.                 labels=labels,
.                 numeric_values=numeric_values,
.                 numeric_values_scale=numeric_values_scale,
.                 float_answer=float_answer,
.             )
.         grads = tape.gradient(outputs.loss, model.trainable_weights)
.         optimizer.apply_gradients(zip(grads, model.trainable_weights))
```
</tf>
</frameworkcontent>

## Usage: inference

<frameworkcontent>
<pt>
Here we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data).


key: _node_content
string_value:
{"id_": "f8449d6d-408d-4d6c-8dab-85cbb0005a65", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d548484a-2102-47a0-9ba8-f41b99635cfe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "2c3c642019f4dea310311c0de08ffd38fc99049d28518bfc687bd6adfeb6e420", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e00cff76-b8df-4477-8bb8-ac7412aafd5f", "node_type": "1", "metadata": {}, "hash": "89dd0ff4dff0907c673ed19305715fa40b1bf3be6c9bccde79f40b78654ec9ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.     init_cell_selection_weights_to_zero=True,
.     select_one_column=True,
.     allow_empty_column_selection=False,
.     temperature=0.0352513,
. )
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)

>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch[0]
.         attention_mask = batch[1]
.         token_type_ids = batch[4]
.         labels = batch[-1]
.         numeric_values = batch[2]
.         numeric_values_scale = batch[3]
.         float_answer = batch[6]

.         # forward + backward + optimize
.         with tf.GradientTape() as tape:
.             outputs = model(
.                 input_ids=input_ids,
.                 attention_mask=attention_mask,
.                 token_type_ids=token_type_ids,
.                 labels=labels,
.                 numeric_values=numeric_values,
.                 numeric_values_scale=numeric_values_scale,
.                 float_answer=float_answer,
.             )
.         grads = tape.gradient(outputs.loss, model.trainable_weights)
.         optimizer.apply_gradients(zip(grads, model.trainable_weights))
```
</tf>
</frameworkcontent>

## Usage: inference

<frameworkcontent>
<pt>
Here we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?


key: _node_content
string_value:
{"id_": "e00cff76-b8df-4477-8bb8-ac7412aafd5f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8449d6d-408d-4d6c-8dab-85cbb0005a65", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "70269f57b62152b6fcb733717729e470bf8f4bfd2b3a0fd735a9eb19ad7b7b96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "91b9ce61-f874-4b7d-b94a-deed97740e3a", "node_type": "1", "metadata": {}, "hash": "9717b0fabfb02fcb3c2ead98d633de276ed56bd1eab76e6e02ce5d15ce051801", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
numeric_values=numeric_values,
.                 numeric_values_scale=numeric_values_scale,
.                 float_answer=float_answer,
.             )
.         grads = tape.gradient(outputs.loss, model.trainable_weights)
.         optimizer.apply_gradients(zip(grads, model.trainable_weights))
```
</tf>
</frameworkcontent>

## Usage: inference

<frameworkcontent>
<pt>
Here we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> outputs = model(**inputs)
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()
... )

>>> # let's print out the results:
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3: "COUNT"}
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]

>>> answers = []
>>> for coordinates in predicted_answer_coordinates:
...     if len(coordinates) == 1:
...         # only a single cell:
...         answers.append(table.iat[coordinates[0]])
...     else:
...         # multiple cells
...         cell_values = []
...         for coordinate in coordinates:
...             cell_values.append(table.iat[coordinate])
...         answers.append(", ".join(cell_values))

>>> display(table)
>>> print("")
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
...     print(query)
...     if predicted_agg == "NONE":
...         print("Predicted answer: " + answer)
...     else:
...         print("Predicted answer: " + predicted_agg + " > " + answer)
What is the name of the first actor?
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</pt>
<tf>
Here we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e.


key: _node_content
string_value:
{"id_": "91b9ce61-f874-4b7d-b94a-deed97740e3a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e00cff76-b8df-4477-8bb8-ac7412aafd5f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "20c2c399cd2ed793df9058b5015b196735f266b6d18c9b7ce1152483cc6d226e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c061f36b-6563-49e7-a513-279dc6d71d4f", "node_type": "1", "metadata": {}, "hash": "16148826af60d7d6f5f0365c8155bc86b92cb18b7174ca5fb72da00db3919f50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30325, "end_char_idx": 33178, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     "What is the total number of movies?",
... ]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> outputs = model(**inputs)
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()
... )

>>> # let's print out the results:
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3: "COUNT"}
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]

>>> answers = []
>>> for coordinates in predicted_answer_coordinates:
...     if len(coordinates) == 1:
...         # only a single cell:
...         answers.append(table.iat[coordinates[0]])
...     else:
...         # multiple cells
...         cell_values = []
...         for coordinate in coordinates:
...             cell_values.append(table.iat[coordinate])
...         answers.append(", ".join(cell_values))

>>> display(table)
>>> print("")
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
...     print(query)
...     if predicted_agg == "NONE":
...         print("Predicted answer: " + answer)
...     else:
...         print("Predicted answer: " + predicted_agg + " > " + answer)
What is the name of the first actor?
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</pt>
<tf>
Here we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?


key: _node_content
string_value:
{"id_": "c061f36b-6563-49e7-a513-279dc6d71d4f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "91b9ce61-f874-4b7d-b94a-deed97740e3a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "42905cb43de7c5c6c7d27340b15d198b991a17a4145339957edfc35cbec2dc2b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c95986f3-a820-4c52-af26-1915152f7e81", "node_type": "1", "metadata": {}, "hash": "22475e8c44e5f9d42ac182183fa5e68edfe42edfaed79b206d56d8a84cf538bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31715, "end_char_idx": 34943, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</pt>
<tf>
Here we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="tf")
>>> outputs = model(**inputs)
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
...     inputs, outputs.logits, outputs.logits_aggregation
... )

>>> # let's print out the results:
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3: "COUNT"}
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]

>>> answers = []
>>> for coordinates in predicted_answer_coordinates:
...     if len(coordinates) == 1:
...         # only a single cell:
...         answers.append(table.iat[coordinates[0]])
...     else:
...         # multiple cells
...         cell_values = []
...         for coordinate in coordinates:
...             cell_values.append(table.iat[coordinate])
...         answers.append(", ".join(cell_values))

>>> display(table)
>>> print("")
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
...     print(query)
...     if predicted_agg == "NONE":
...         print("Predicted answer: " + answer)
...     else:
...         print("Predicted answer: " + predicted_agg + " > " + answer)
What is the name of the first actor?
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</tf>
</frameworkcontent>

In case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair.


key: _node_content
string_value:
{"id_": "c95986f3-a820-4c52-af26-1915152f7e81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3dcd76eb-e958-488d-a887-8d0ff2acc337", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c061f36b-6563-49e7-a513-279dc6d71d4f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "e58991972fc6735dfa3b608f3b432ebba2d704a9573b469d180f1eebec0481d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34503, "end_char_idx": 36359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</tf>
</frameworkcontent>

In case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair. Again, more info can be found in [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for PyTorch) and [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for TensorFlow).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Masked language modeling task guide](../tasks/masked_language_modeling)

## TAPAS specific outputs
[[autodoc]] models.tapas.modeling_tapas.TableQuestionAnsweringOutput

## TapasConfig
[[autodoc]] TapasConfig

## TapasTokenizer
[[autodoc]] TapasTokenizer
    - __call__
    - convert_logits_to_predictions
    - save_vocabulary

<frameworkcontent>
<pt>

## TapasModel
[[autodoc]] TapasModel
    - forward
    
## TapasForMaskedLM
[[autodoc]] TapasForMaskedLM
    - forward

## TapasForSequenceClassification
[[autodoc]] TapasForSequenceClassification
    - forward
    
## TapasForQuestionAnswering
[[autodoc]] TapasForQuestionAnswering
    - forward

</pt>
<tf>

## TFTapasModel
[[autodoc]] TFTapasModel
    - call
    
## TFTapasForMaskedLM
[[autodoc]] TFTapasForMaskedLM
    - call

## TFTapasForSequenceClassification
[[autodoc]] TFTapasForSequenceClassification
    - call
    
## TFTapasForQuestionAnswering
[[autodoc]] TFTapasForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "7c42cc0d-0303-46c7-9cf2-d41f7fe242c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2964f30a-2ced-436c-9857-9e20d5973d85", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "hash": "824354140f99d988dc7a66f6a4824d0281e10ec39c4c3654187d84104ce96d51", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da1e3988-8aab-4052-9073-b86d8421fa8a", "node_type": "1", "metadata": {}, "hash": "e564a0391f04e2e1af5b3fcec19d1145d103f78db3cebc46031ee9100fda3cb6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3312, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LXMERT

## Overview

The LXMERT model was proposed in [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://arxiv.org/abs/1908.07490) by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders
(one for the vision modality, one for the language modality, and then one to fuse both modalities) pretrained using a
combination of masked language modeling, visual-language text alignment, ROI-feature regression, masked
visual-attribute modeling, masked visual-object modeling, and visual-question answering objectives. The pretraining
consists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering, VQA 2.0, and GQA.

The abstract from the paper is the following:

*Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly,
the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality
Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we
build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language
encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language
semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative
pretraining tasks: masked language modeling, masked object prediction (feature regression and label classification),
cross-modality matching, and image question answering. These tasks help in learning both intra-modality and
cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art
results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our
pretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous
best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel
model components and pretraining strategies significantly contribute to our strong results; and also present several
attention visualizations for the different encoders*

This model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).

## Usage tips

- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features
  will work.
- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the
  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to
  itself, select the vision/language hidden states from the first input in the tuple.
- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used
  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder
  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and
  both self attention outputs are disregarded.


key: _node_content
string_value:
{"id_": "da1e3988-8aab-4052-9073-b86d8421fa8a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2964f30a-2ced-436c-9857-9e20d5973d85", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "hash": "824354140f99d988dc7a66f6a4824d0281e10ec39c4c3654187d84104ce96d51", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c42cc0d-0303-46c7-9cf2-d41f7fe242c3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "hash": "a80a91485425c84cebe9bbc35b31447b0cf52cdaaad664256041bce0637452ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1618, "end_char_idx": 4322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These tasks help in learning both intra-modality and
cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art
results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our
pretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous
best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel
model components and pretraining strategies significantly contribute to our strong results; and also present several
attention visualizations for the different encoders*

This model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).

## Usage tips

- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features
  will work.
- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the
  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to
  itself, select the vision/language hidden states from the first input in the tuple.
- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used
  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder
  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and
  both self attention outputs are disregarded.

## Resources

- [Question answering task guide](../tasks/question_answering)

## LxmertConfig

[[autodoc]] LxmertConfig

## LxmertTokenizer

[[autodoc]] LxmertTokenizer

## LxmertTokenizerFast

[[autodoc]] LxmertTokenizerFast

## Lxmert specific outputs

[[autodoc]] models.lxmert.modeling_lxmert.LxmertModelOutput

[[autodoc]] models.lxmert.modeling_lxmert.LxmertForPreTrainingOutput

[[autodoc]] models.lxmert.modeling_lxmert.LxmertForQuestionAnsweringOutput

[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertModelOutput

[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertForPreTrainingOutput

<frameworkcontent>
<pt>

## LxmertModel

[[autodoc]] LxmertModel
    - forward

## LxmertForPreTraining

[[autodoc]] LxmertForPreTraining
    - forward

## LxmertForQuestionAnswering

[[autodoc]] LxmertForQuestionAnswering
    - forward

</pt>
<tf>

## TFLxmertModel

[[autodoc]] TFLxmertModel
    - call

## TFLxmertForPreTraining

[[autodoc]] TFLxmertForPreTraining
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "0a23cbde-495b-42e9-9682-0f5a39b4ae89", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "04104621-9d10-40fb-a0ae-896a8147dcf0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "hash": "17fd551fd712ca727ea3d1a97dc6e44926a8ce1ac6daf07d80f77850878fa9f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6ba5719-2652-4e00-b0d0-17f7dc57e207", "node_type": "1", "metadata": {}, "hash": "ef621a0e76d8e3fd4223cefff8da8a75cce207dd7f7212953fc6e8e55d23a5c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3416, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DPT

## Overview

The DPT model was proposed in [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by René Ranftl, Alexey Bochkovskiy, Vladlen Koltun.
DPT is a model that leverages the [Vision Transformer (ViT)](vit) as backbone for dense prediction tasks like semantic segmentation and depth estimation.

The abstract from the paper is the following:

*We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg"
alt="drawing" width="600"/>

<small> DPT architecture. Taken from the <a href="https://arxiv.org/abs/2103.13413" target="_blank">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).

## Usage tips

DPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:

```python
from transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation

# initialize with a Transformer-based backbone such as DINOv2
# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)
backbone_config = Dinov2Config.from_pretrained("facebook/dinov2-base", out_features=["stage1", "stage2", "stage3", "stage4"], reshape_hidden_states=False)

config = DPTConfig(backbone_config=backbone_config)
model = DPTForDepthEstimation(config=config)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DPT.

- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).

- [Semantic segmentation task guide](../tasks/semantic_segmentation)
- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!


key: _node_content
string_value:
{"id_": "f6ba5719-2652-4e00-b0d0-17f7dc57e207", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "04104621-9d10-40fb-a0ae-896a8147dcf0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "hash": "17fd551fd712ca727ea3d1a97dc6e44926a8ce1ac6daf07d80f77850878fa9f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a23cbde-495b-42e9-9682-0f5a39b4ae89", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "hash": "70a20fe5895b116c67d4d36ee3bfe27be499f9c990e4b9bef75cc8c8f33fc4b9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1941, "end_char_idx": 3985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).

## Usage tips

DPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:

```python
from transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation

# initialize with a Transformer-based backbone such as DINOv2
# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)
backbone_config = Dinov2Config.from_pretrained("facebook/dinov2-base", out_features=["stage1", "stage2", "stage3", "stage4"], reshape_hidden_states=False)

config = DPTConfig(backbone_config=backbone_config)
model = DPTForDepthEstimation(config=config)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DPT.

- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).

- [Semantic segmentation task guide](../tasks/semantic_segmentation)
- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DPTConfig

[[autodoc]] DPTConfig

## DPTFeatureExtractor

[[autodoc]] DPTFeatureExtractor
    - __call__
    - post_process_semantic_segmentation

## DPTImageProcessor

[[autodoc]] DPTImageProcessor
    - preprocess
    - post_process_semantic_segmentation

## DPTModel

[[autodoc]] DPTModel
    - forward

## DPTForDepthEstimation

[[autodoc]] DPTForDepthEstimation
    - forward

## DPTForSemanticSegmentation

[[autodoc]] DPTForSemanticSegmentation
    - forward


key: _node_content
string_value:
{"id_": "bbf978b9-0ec4-475d-bd3f-e85719235b14", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "28747e56-6f81-4f9d-a9bc-a4b9af6fbdd6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "hash": "f59dcff97f3e1d243275ecb5e64ce123396e3adbfb60ef6e63c92dba3da20e78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "629a9ceb-2021-4b68-90ea-9da395e507d5", "node_type": "1", "metadata": {}, "hash": "b1d5a869de85b830a05286ec1e0ae4567c892fefbab81eece9fb43d3f749df7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3319, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CANINE

## Overview

The CANINE model was proposed in [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language
Representation](https://arxiv.org/abs/2103.06874) by Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting. It's
among the first papers that trains a Transformer without using an explicit tokenization step (such as Byte Pair
Encoding (BPE), WordPiece or SentencePiece). Instead, the model is trained directly at a Unicode character-level.
Training at a character-level inevitably comes with a longer sequence length, which CANINE solves with an efficient
downsampling strategy, before applying a deep Transformer encoder.

The abstract from the paper is the following:

*Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models
still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword
lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all
languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE,
a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a
pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias.
To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input
sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by
2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/language/tree/master/language/canine).

## Usage tips

- CANINE uses no less than 3 Transformer encoders internally: 2 "shallow" encoders (which only consist of a single
  layer) and 1 "deep" encoder (which is a regular BERT encoder). First, a "shallow" encoder is used to contextualize
  the character embeddings, using local attention. Next, after downsampling, a "deep" encoder is applied. Finally,
  after upsampling, a "shallow" encoder is used to create the final character embeddings. Details regarding up- and
  downsampling can be found in the paper.
- CANINE uses a max sequence length of 2048 characters by default. One can use [`CanineTokenizer`]
  to prepare text for the model.
- Classification can be done by placing a linear layer on top of the final hidden state of the special [CLS] token
  (which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of
  tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The
  details for this can be found in the paper.

Model checkpoints:

  - [google/canine-c](https://huggingface.co/google/canine-c): Pre-trained with autoregressive character loss,
    12-layer, 768-hidden, 12-heads, 121M parameters (size ~500 MB).
  - [google/canine-s](https://huggingface.co/google/canine-s): Pre-trained with subword loss, 12-layer,
    768-hidden, 12-heads, 121M parameters (size ~500 MB).


key: _node_content
string_value:
{"id_": "629a9ceb-2021-4b68-90ea-9da395e507d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "28747e56-6f81-4f9d-a9bc-a4b9af6fbdd6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "hash": "f59dcff97f3e1d243275ecb5e64ce123396e3adbfb60ef6e63c92dba3da20e78", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bbf978b9-0ec4-475d-bd3f-e85719235b14", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "hash": "4c4d6c9980cba714dac6e501bec0e2b2a66ba7b9107ec4d47a96e2ec585f0ca7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3322, "end_char_idx": 5566, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage example

CANINE works on raw characters, so it can be used **without a tokenizer**:

```python
>>> from transformers import CanineModel
>>> import torch

>>> model = CanineModel.from_pretrained("google/canine-c")  # model pre-trained with autoregressive character loss

>>> text = "hello world"
>>> # use Python's built-in ord() function to turn each character into its unicode code point id
>>> input_ids = torch.tensor([[ord(char) for char in text]])

>>> outputs = model(input_ids)  # forward pass
>>> pooled_output = outputs.pooler_output
>>> sequence_output = outputs.last_hidden_state
```

For batched inference and training, it is however recommended to make use of the tokenizer (to pad/truncate all
sequences to the same length):

```python
>>> from transformers import CanineTokenizer, CanineModel

>>> model = CanineModel.from_pretrained("google/canine-c")
>>> tokenizer = CanineTokenizer.from_pretrained("google/canine-c")

>>> inputs = ["Life is like a box of chocolates.", "You never know what you gonna get."]
>>> encoding = tokenizer(inputs, padding="longest", truncation=True, return_tensors="pt")

>>> outputs = model(**encoding)  # forward pass
>>> pooled_output = outputs.pooler_output
>>> sequence_output = outputs.last_hidden_state
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Multiple choice task guide](../tasks/multiple_choice)

## CanineConfig

[[autodoc]] CanineConfig

## CanineTokenizer

[[autodoc]] CanineTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences

## CANINE specific outputs

[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling

## CanineModel

[[autodoc]] CanineModel
    - forward

## CanineForSequenceClassification

[[autodoc]] CanineForSequenceClassification
    - forward

## CanineForMultipleChoice

[[autodoc]] CanineForMultipleChoice
    - forward

## CanineForTokenClassification

[[autodoc]] CanineForTokenClassification
    - forward

## CanineForQuestionAnswering

[[autodoc]] CanineForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "d082da5d-90be-40e8-a991-807826f9992d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9804c541-443c-43a1-a508-0d344abcf22d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c5d21a3-2d06-4767-ac74-93b460535df1", "node_type": "1", "metadata": {}, "hash": "93be05c5d0cf367ff6ed445162b83a659557faa32e35e1cf703f9b6d6bf38f38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1708, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VipLlava

## Overview

The VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.

VipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a "red bounding box" or "pointed arrow" during training.

The abstract from the paper is the following:

*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*

The original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)


key: _node_content
string_value:
{"id_": "2c5d21a3-2d06-4767-ac74-93b460535df1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9804c541-443c-43a1-a508-0d344abcf22d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d082da5d-90be-40e8-a991-807826f9992d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "25d652330a465a6dfb45458464a53c5e431e16d85d8a076d195846de4f02915f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "31e56080-5796-458d-bd08-025db34d50a6", "node_type": "1", "metadata": {}, "hash": "98ee1174bd6aac28765f57e81130931612ca885aaa5c686caabeb6a339670a2f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4336, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VipLlava

## Overview

The VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.

VipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a "red bounding box" or "pointed arrow" during training.

The abstract from the paper is the following:

*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*

The original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)


## Usage tips:

- The architecture is similar than llava architecture except that the multi-modal projector takes a set of concatenated vision hidden states and has an additional layernorm layer on that module.

- We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating.

- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.

- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys "role" and "content". The "content" should be a list of dictionaries, for "text" and "image" modalities, as follows:

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What’s shown in this image?"},
         ,
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image in more details."},
        ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
>>> "###Human: <image>\nWhat’s shown in this image?###Assistant: This image shows a red stop sign.###Human: Describe the image in more details.###Assistant:"
```

- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by VipLLaVa checkpoints:
```bash
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\n<prompt>###Assistant:
```

For multiple turns conversation:
```bash
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\n<prompt1>###Assistant: <answer1>###Human: <prompt2>###Assistant:
```


key: _node_content
string_value:
{"id_": "31e56080-5796-458d-bd08-025db34d50a6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9804c541-443c-43a1-a508-0d344abcf22d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c5d21a3-2d06-4767-ac74-93b460535df1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "ed15255c288e104d76811a249adbd344afd2c5b4157010b6205ea37e86b444f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4339, "end_char_idx": 4481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## VipLlavaConfig

[[autodoc]] VipLlavaConfig

## VipLlavaForConditionalGeneration

[[autodoc]] VipLlavaForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "def19be9-b52e-492b-a9ea-3057a00eef7d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c11219b4-07a1-4b43-9e2e-4ac20938d754", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "77d1eaf82000b06e3e53eee67dc9a32103714fcff59b43350da89e924109c025", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8bbe4767-349f-4bb2-ac46-9a2f34b29b72", "node_type": "1", "metadata": {}, "hash": "4999b2128a4caac8a2de2ceb2add6f3c9773df08a0b09e6befa9d18d535a379a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# NLLB

## Updated tokenizer behavior 

**DISCLAIMER:** The default behaviour for the tokenizer was fixed and thus changed in April 2023.
The previous version adds `[self.eos_token_id, self.cur_lang_code]` at the end of the token sequence for both target and source tokenization. This is wrong as the NLLB paper mentions (page 48, 6.1.1. Model Architecture) :

*Note that we prefix the source sequence with the source language, as opposed to the target
language as previously done in several works (Arivazhagan et al., 2019; Johnson et al.,
2017). This is primarily because we prioritize optimizing zero-shot performance of our
model on any pair of 200 languages at a minor cost to supervised performance.*

Previous behaviour:

```python
>>> from transformers import NllbTokenizer

>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer("How was your day?").input_ids
[13374, 1398, 4260, 4039, 248130, 2, 256047]

>>> # 2: '</s>'
>>> # 256047 : 'eng_Latn'
```
New behaviour

```python
>>> from transformers import NllbTokenizer

>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer("How was your day?").input_ids
[256047, 13374, 1398, 4260, 4039, 248130, 2]
 ```

Enabling the old behaviour can be done as follows:
```python
>>> from transformers import NllbTokenizer

>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", legacy_behaviour=True)
```

For more details, feel free to check the linked [PR](https://github.com/huggingface/transformers/pull/22313) and [Issue](https://github.com/huggingface/transformers/issues/19943).

## Overview

The NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-jussà, James Cross, Onur Çelebi,
Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,
Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,
Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Safiyyah Saleem, Holger Schwenk, and Jeff Wang.

The abstract of the paper is the following:

*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the
200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind?


key: _node_content
string_value:
{"id_": "8bbe4767-349f-4bb2-ac46-9a2f34b29b72", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c11219b4-07a1-4b43-9e2e-4ac20938d754", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "77d1eaf82000b06e3e53eee67dc9a32103714fcff59b43350da89e924109c025", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "def19be9-b52e-492b-a9ea-3057a00eef7d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "069e487edb67c2ba49361b3cc923ef8ddf78f05883e7a66e0a11ad69fd3c1d15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5678d84-7f49-4504-8451-4a2fee9f8a15", "node_type": "1", "metadata": {}, "hash": "f33381fbfb3156308d83d230c7eff209467053e762dfd06347dc418cae4c7aba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1645, "end_char_idx": 4778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Overview

The NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-jussà, James Cross, Onur Çelebi,
Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,
Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,
Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Safiyyah Saleem, Holger Schwenk, and Jeff Wang.

The abstract of the paper is the following:

*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the
200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by
first contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed
at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of
Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training
improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using
a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.
Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*

This implementation contains the dense models available on release.

**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**

This model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).

## Generating with NLLB

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.


key: _node_content
string_value:
{"id_": "c5678d84-7f49-4504-8451-4a2fee9f8a15", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c11219b4-07a1-4b43-9e2e-4ac20938d754", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "77d1eaf82000b06e3e53eee67dc9a32103714fcff59b43350da89e924109c025", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8bbe4767-349f-4bb2-ac46-9a2f34b29b72", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "599e06f839ed76580ce020084de6d114d2ad4e9de092ce5c268f22d467cbb9a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe63bac3-4e2e-43eb-9792-545266e61302", "node_type": "1", "metadata": {}, "hash": "4df3401664828a832c781cd7faf867ac0158deb91063508ade66a5edd6ca4a08", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3089, "end_char_idx": 5742, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then, we created datasets and models aimed
at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of
Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training
improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using
a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.
Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*

This implementation contains the dense models available on release.

**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**

This model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).

## Generating with NLLB

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "UN Chief says there is no military solution in Syria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["fra_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
Le chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie
```

### Generating from any other language than English

English (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.


key: _node_content
string_value:
{"id_": "fe63bac3-4e2e-43eb-9792-545266e61302", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c11219b4-07a1-4b43-9e2e-4ac20938d754", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "77d1eaf82000b06e3e53eee67dc9a32103714fcff59b43350da89e924109c025", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5678d84-7f49-4504-8451-4a2fee9f8a15", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "c4a7e081882a62bda43b627d54f70867f76756ef18f906bbe430c4e307f00059", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48774d0d-78cd-4e91-a594-007b7bb84564", "node_type": "1", "metadata": {}, "hash": "af21da8b814c9356ad6a44eedb0cf51f33238f8a9b613e88fd911e4ec1a3473f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4229, "end_char_idx": 7157, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).

## Generating with NLLB

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "UN Chief says there is no military solution in Syria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["fra_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
Le chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie
```

### Generating from any other language than English

English (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.

See example below for a translation from romanian to german:

```py
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained(
...     "facebook/nllb-200-distilled-600M", token=True, src_lang="ron_Latn"
... )
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", token=True)

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
UN-Chef sagt, es gibt keine militärische Lösung in Syrien
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## NllbTokenizer

[[autodoc]] NllbTokenizer
    - build_inputs_with_special_tokens

## NllbTokenizerFast

[[autodoc]] NllbTokenizerFast

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).


key: _node_content
string_value:
{"id_": "48774d0d-78cd-4e91-a594-007b7bb84564", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c11219b4-07a1-4b43-9e2e-4ac20938d754", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "77d1eaf82000b06e3e53eee67dc9a32103714fcff59b43350da89e924109c025", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe63bac3-4e2e-43eb-9792-545266e61302", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "352eabb2cf47f28e2bae02338761f53f707f0b8cde797c4ea74a5b9424c954ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03797b7a-1a29-401f-86c5-467211fa0314", "node_type": "1", "metadata": {}, "hash": "0460f70ad625e204b926b12d4e75d67ae849511bbb326a0921b91d7e5b21d869", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5564, "end_char_idx": 8592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.

See example below for a translation from romanian to german:

```py
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained(
...     "facebook/nllb-200-distilled-600M", token=True, src_lang="ron_Latn"
... )
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", token=True)

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
UN-Chef sagt, es gibt keine militärische Lösung in Syrien
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## NllbTokenizer

[[autodoc]] NllbTokenizer
    - build_inputs_with_special_tokens

## NllbTokenizerFast

[[autodoc]] NllbTokenizerFast

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.

```python
>>> import torch
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda").eval()
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt").to("cuda")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"UN-Chef sagt, es gibt keine militärische Lösung in Syrien"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.


key: _node_content
string_value:
{"id_": "03797b7a-1a29-401f-86c5-467211fa0314", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c11219b4-07a1-4b43-9e2e-4ac20938d754", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "77d1eaf82000b06e3e53eee67dc9a32103714fcff59b43350da89e924109c025", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48774d0d-78cd-4e91-a594-007b7bb84564", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "35bbe9f1d69ee178cc79a671af5272dea84e53465ef3bfdf3b04a4da1fba2b50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7001, "end_char_idx": 8740, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.

```python
>>> import torch
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda").eval()
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt").to("cuda")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"UN-Chef sagt, es gibt keine militärische Lösung in Syrien"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp">
</div>


key: _node_content
string_value:
{"id_": "98bfe556-710c-4a8c-9c52-e6f5c176f593", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6add301a-2c7c-4c58-96c1-37a0bacc6dbb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbe975a8-c53c-4161-8a00-fed8eef16f76", "node_type": "1", "metadata": {}, "hash": "f39a0e5529512db7b56c4c5e69f1d32a58ab45bc145f1178d7a17b01bf978729", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3321, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DeiT

## Overview

The DeiT model was proposed in [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, Hervé Jégou. The [Vision Transformer (ViT)](vit) introduced in [Dosovitskiy et al., 2020](https://arxiv.org/abs/2010.11929) has shown that one can match or even outperform existing convolutional neural
networks using a Transformer encoder (BERT-like). However, the ViT models introduced in that paper required training on
expensive infrastructure for multiple weeks, using external data. DeiT (data-efficient image transformers) are more
efficiently trained transformers for image classification, requiring far less data and far less computing resources
compared to the original ViT models.

The abstract from the paper is the following:

*Recently, neural networks purely based on attention were shown to address image understanding tasks such as image
classification. However, these visual transformers are pre-trained with hundreds of millions of images using an
expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free
transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision
transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external
data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation
token ensuring that the student learns from the teacher through attention. We show the interest of this token-based
distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets
for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and
models.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).

## Usage tips

- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the
  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with
  the class ([CLS]) and patch tokens through the self-attention layers.
- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top
  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a
  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction
  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the
  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the
  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction
  between both heads as final prediction. (2) is also called "fine-tuning with distillation", because one relies on a
  teacher that has already been fine-tuned on the downstream dataset.


key: _node_content
string_value:
{"id_": "dbe975a8-c53c-4161-8a00-fed8eef16f76", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6add301a-2c7c-4c58-96c1-37a0bacc6dbb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98bfe556-710c-4a8c-9c52-e6f5c176f593", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "441e3e7de384fe5762793e113bab4cd0119e40f4193b67866e01a082afa87d59", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4a4dfe7-89fb-42ce-936c-71dcdb14c13e", "node_type": "1", "metadata": {}, "hash": "8900adc0324e373391c6583eb4712e15c41d06e2414db67c22b253bde705edbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1662, "end_char_idx": 4842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We show the interest of this token-based
distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets
for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and
models.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).

## Usage tips

- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the
  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with
  the class ([CLS]) and patch tokens through the self-attention layers.
- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top
  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a
  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction
  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the
  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the
  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction
  between both heads as final prediction. (2) is also called "fine-tuning with distillation", because one relies on a
  teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds to
  [`DeiTForImageClassification`] and (2) corresponds to
  [`DeiTForImageClassificationWithTeacher`].
- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is
  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.
- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in
  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for
  pre-training.
- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into
  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data
  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset
  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):
  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and
  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to
  prepare images for the model.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use.


key: _node_content
string_value:
{"id_": "a4a4dfe7-89fb-42ce-936c-71dcdb14c13e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6add301a-2c7c-4c58-96c1-37a0bacc6dbb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dbe975a8-c53c-4161-8a00-fed8eef16f76", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "a61caf16365eadfd663d915a8e69dfb8fccaf9e99ea1b4c687b852fe55c757e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f2f32104-8001-4fc0-b5cb-188f94af3ce1", "node_type": "1", "metadata": {}, "hash": "029ca90e3a085074fd8493a422be77182cdb82522184983bf5b3107400b982dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3322, "end_char_idx": 5839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In terms of models, (1) corresponds to
  [`DeiTForImageClassification`] and (2) corresponds to
  [`DeiTForImageClassificationWithTeacher`].
- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is
  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.
- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in
  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for
  pre-training.
- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into
  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data
  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset
  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):
  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and
  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to
  prepare images for the model.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import DeiTForImageClassification
model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.


key: _node_content
string_value:
{"id_": "f2f32104-8001-4fc0-b5cb-188f94af3ce1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6add301a-2c7c-4c58-96c1-37a0bacc6dbb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4a4dfe7-89fb-42ce-936c-71dcdb14c13e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "667d94f09a853d5e030af32ed35b1fed8b792b55bbc22602707dafaa533d9d5b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ad74a5b-c083-40fb-b158-473ccc208e19", "node_type": "1", "metadata": {}, "hash": "48d04040b93eee91b93afc81ec5bba08c29cfde39234bf8ac7a8f2bfa36633dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4478, "end_char_idx": 7589, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that one should use [`DeiTImageProcessor`] in order to
  prepare images for the model.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import DeiTForImageClassification
model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         8 |                                         6 |                      1.33 |
|            2 |                                         9 |                                         6 |                      1.5  |
|            4 |                                         9 |                                         6 |                      1.5  |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DeiT.

<PipelineTag pipeline="image-classification"/>

- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

Besides that:

- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "2ad74a5b-c083-40fb-b158-473ccc208e19", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6add301a-2c7c-4c58-96c1-37a0bacc6dbb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f2f32104-8001-4fc0-b5cb-188f94af3ce1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cd83813f908408df4533323524bdc6c90912a0f6c096b10602640297998e3b62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5658, "end_char_idx": 8498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         8 |                                         6 |                      1.33 |
|            2 |                                         9 |                                         6 |                      1.5  |
|            4 |                                         9 |                                         6 |                      1.5  |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DeiT.

<PipelineTag pipeline="image-classification"/>

- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

Besides that:

- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DeiTConfig

[[autodoc]] DeiTConfig

## DeiTFeatureExtractor

[[autodoc]] DeiTFeatureExtractor
    - __call__

## DeiTImageProcessor

[[autodoc]] DeiTImageProcessor
    - preprocess

<frameworkcontent>
<pt>

## DeiTModel

[[autodoc]] DeiTModel
    - forward

## DeiTForMaskedImageModeling

[[autodoc]] DeiTForMaskedImageModeling
    - forward

## DeiTForImageClassification

[[autodoc]] DeiTForImageClassification
    - forward

## DeiTForImageClassificationWithTeacher

[[autodoc]] DeiTForImageClassificationWithTeacher
    - forward

</pt>
<tf>

## TFDeiTModel

[[autodoc]] TFDeiTModel
    - call

## TFDeiTForMaskedImageModeling

[[autodoc]] TFDeiTForMaskedImageModeling
    - call

## TFDeiTForImageClassification

[[autodoc]] TFDeiTForImageClassification
    - call

## TFDeiTForImageClassificationWithTeacher

[[autodoc]] TFDeiTForImageClassificationWithTeacher
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "2e039595-c802-4a04-ab0b-dc7e0bf97b81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pegasus_x", "title": "PEGASUS-X", "tokens": 630, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "33319b3d-6907-44eb-a807-88d10507d3b5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pegasus_x", "title": "PEGASUS-X", "tokens": 630, "retrieve_doc": true, "source": "transformers"}, "hash": "2ecb7f0c047d782b936dfcdeac84b858c4d7fa5cd16967f6d4028bd9be2f1789", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2134, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PEGASUS-X

## Overview

The PEGASUS-X model was proposed in [Investigating Efficiently Extending Transformers for Long Input Summarization](https://arxiv.org/abs/2208.04347)  by Jason Phang, Yao Zhao and Peter J. Liu.

PEGASUS-X (PEGASUS eXtended) extends the PEGASUS models for long input summarization through additional long input pretraining and using staggered block-local attention with global tokens in the encoder.

The abstract from the paper is the following:

*While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most pretrained models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms can most efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. PEGASUS-X achieves strong performance on long input summarization tasks comparable with much larger models while adding few additional parameters and not requiring model parallelism to train.*

This model was contributed by [zphang](https://huggingface.co/zphang). The original code can be found [here](https://github.com/google-research/pegasus).

## Documentation resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

<Tip>

PEGASUS-X uses the same tokenizer as [PEGASUS](pegasus).

</Tip>

## PegasusXConfig

[[autodoc]] PegasusXConfig

## PegasusXModel

[[autodoc]] PegasusXModel
    - forward

## PegasusXForConditionalGeneration

[[autodoc]] PegasusXForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "cf663f87-8036-4e7c-91e4-b7e2a265fd25", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "694fd601-aba0-4349-a965-ebd4215ba034", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "455f5cfc-69d0-4d2e-9b72-0baa9c8844e6", "node_type": "1", "metadata": {}, "hash": "65d57d19aec87f1f8f24b08ef79d17237774788559ef0021cc1da7a25f288e53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3857, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Wav2Vec2-BERT

## Overview

The Wav2Vec2-BERT model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.

This model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification.

The official results of the model can be found in Section 3.2.1 of the paper.

The abstract from the paper is the following:

*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model—SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one’s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work—including models, code, and a watermark detector—are publicly released and accessible at the link below.*

This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## Usage tips

- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.


key: _node_content
string_value:
{"id_": "455f5cfc-69d0-4d2e-9b72-0baa9c8844e6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "694fd601-aba0-4349-a965-ebd4215ba034", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf663f87-8036-4e7c-91e4-b7e2a265fd25", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "2b0d106a5362f899f9b16eebdff7cc010e29d965dcb6f6e925ef321302577821", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc22537a-1bde-4df4-9be6-5eb6ae12871b", "node_type": "1", "metadata": {}, "hash": "2e633c2b946c447014c7daa89f3b7aabd29d91ce849309fba975fbcf4b232c52", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1906, "end_char_idx": 5121, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work—including models, code, and a watermark detector—are publicly released and accessible at the link below.*

This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## Usage tips

- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.
- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or
  rotary position embeddings by setting the correct `config.position_embeddings_type`.
- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.

## Resources

<PipelineTag pipeline="automatic-speech-recognition"/>

- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).
- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).

<PipelineTag pipeline="audio-classification"/>

- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).
- See also: [Audio classification task guide](../tasks/audio_classification)


key: _node_content
string_value:
{"id_": "fc22537a-1bde-4df4-9be6-5eb6ae12871b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "694fd601-aba0-4349-a965-ebd4215ba034", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "455f5cfc-69d0-4d2e-9b72-0baa9c8844e6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "7dff3bf64c6c2812e1f5de426679ae9d826cc48c860730fff49a94583e87dab1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3452, "end_char_idx": 5770, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## Usage tips

- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.
- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or
  rotary position embeddings by setting the correct `config.position_embeddings_type`.
- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.

## Resources

<PipelineTag pipeline="automatic-speech-recognition"/>

- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).
- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).

<PipelineTag pipeline="audio-classification"/>

- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).
- See also: [Audio classification task guide](../tasks/audio_classification)


## Wav2Vec2BertConfig

[[autodoc]] Wav2Vec2BertConfig

## Wav2Vec2BertProcessor

[[autodoc]] Wav2Vec2BertProcessor
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## Wav2Vec2BertModel

[[autodoc]] Wav2Vec2BertModel
    - forward

## Wav2Vec2BertForCTC

[[autodoc]] Wav2Vec2BertForCTC
    - forward

## Wav2Vec2BertForSequenceClassification

[[autodoc]] Wav2Vec2BertForSequenceClassification
    - forward

## Wav2Vec2BertForAudioFrameClassification

[[autodoc]] Wav2Vec2BertForAudioFrameClassification
    - forward

## Wav2Vec2BertForXVector

[[autodoc]] Wav2Vec2BertForXVector
    - forward


key: _node_content
string_value:
{"id_": "24c1d159-bcd1-4b3f-aecc-16ceaa569b1b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1002cdec-ab8b-4b56-8637-a9412a5ae6bf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "hash": "17479d3a217cd878a98f8600f35ecc4d0c3d51fd8eb0402d29764483afabb83b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56d1d2b6-4bbc-4128-90bd-b1032778f4be", "node_type": "1", "metadata": {}, "hash": "ffab6bcf96998c282e8a43a358cf36503f958dc381f5c1735009fae9a74a59e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1700, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoFormer

## Overview

The RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.

The abstract from the paper is the following:

*Position encoding in transformer architecture provides supervision for dependency modeling between elements at
different positions in the sequence. We investigate various methods to encode positional information in
transformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The
proposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative
position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of
being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and
capability of equipping the linear self-attention with relative position encoding. As a result, the enhanced
transformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We
release the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing
experiment for English benchmark will soon be updated.*

This model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).

## Usage tips
RoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown 
improved performance on classification tasks with long texts.


key: _node_content
string_value:
{"id_": "56d1d2b6-4bbc-4128-90bd-b1032778f4be", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1002cdec-ab8b-4b56-8637-a9412a5ae6bf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "hash": "17479d3a217cd878a98f8600f35ecc4d0c3d51fd8eb0402d29764483afabb83b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24c1d159-bcd1-4b3f-aecc-16ceaa569b1b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "hash": "b7834f109bb1166e071a7b3456f8211ab4d7fa7a96438056e035befca1299ddb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4211, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoFormer

## Overview

The RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.

The abstract from the paper is the following:

*Position encoding in transformer architecture provides supervision for dependency modeling between elements at
different positions in the sequence. We investigate various methods to encode positional information in
transformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The
proposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative
position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of
being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and
capability of equipping the linear self-attention with relative position encoding. As a result, the enhanced
transformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We
release the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing
experiment for English benchmark will soon be updated.*

This model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).

## Usage tips
RoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown 
improved performance on classification tasks with long texts.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## RoFormerConfig

[[autodoc]] RoFormerConfig

## RoFormerTokenizer

[[autodoc]] RoFormerTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## RoFormerTokenizerFast

[[autodoc]] RoFormerTokenizerFast
    - build_inputs_with_special_tokens

<frameworkcontent>
<pt>

## RoFormerModel

[[autodoc]] RoFormerModel
    - forward

## RoFormerForCausalLM

[[autodoc]] RoFormerForCausalLM
    - forward

## RoFormerForMaskedLM

[[autodoc]] RoFormerForMaskedLM
    - forward

## RoFormerForSequenceClassification

[[autodoc]] RoFormerForSequenceClassification
    - forward

## RoFormerForMultipleChoice

[[autodoc]] RoFormerForMultipleChoice
    - forward

## RoFormerForTokenClassification

[[autodoc]] RoFormerForTokenClassification
    - forward

## RoFormerForQuestionAnswering

[[autodoc]] RoFormerForQuestionAnswering
    - forward

</pt>
<tf>

## TFRoFormerModel

[[autodoc]] TFRoFormerModel
    - call

## TFRoFormerForMaskedLM

[[autodoc]] TFRoFormerForMaskedLM
    - call

## TFRoFormerForCausalLM

[[autodoc]] TFRoFormerForCausalLM
    - call

## TFRoFormerForSequenceClassification

[[autodoc]] TFRoFormerForSequenceClassification
    - call

## TFRoFormerForMultipleChoice

[[autodoc]] TFRoFormerForMultipleChoice
    - call

## TFRoFormerForTokenClassification

[[autodoc]] TFRoFormerForTokenClassification
    - call

## TFRoFormerForQuestionAnswering

[[autodoc]] TFRoFormerForQuestionAnswering
    - call

</tf>
<jax>

## FlaxRoFormerModel

[[autodoc]] FlaxRoFormerModel
    - __call__

## FlaxRoFormerForMaskedLM

[[autodoc]] FlaxRoFormerForMaskedLM
    - __call__

## FlaxRoFormerForSequenceClassification

[[autodoc]] FlaxRoFormerForSequenceClassification
    - __call__

## FlaxRoFormerForMultipleChoice

[[autodoc]] FlaxRoFormerForMultipleChoice
    - __call__

## FlaxRoFormerForTokenClassification

[[autodoc]] FlaxRoFormerForTokenClassification
    - __call__

## FlaxRoFormerForQuestionAnswering

[[autodoc]] FlaxRoFormerForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "77d1e619-da76-417e-8d4c-3c48d5c49e86", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5ac55ae9-690b-46b5-84eb-a9228074f540", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8f8c34f7-4b77-4a94-8b17-481b3b15ab81", "node_type": "1", "metadata": {}, "hash": "81e302000b73b7f8ce2ec8be03622b6551b8faebac0a5553ec089d84c0e61ac6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3421, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OWL-ViT

## Overview

The OWL-ViT (short for Vision Transformer for Open-World Localization) was proposed in [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230) by Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. OWL-ViT is an open-vocabulary object detection network trained on a variety of (image, text) pairs. It can be used to query an image with one or multiple text queries to search for and detect target objects described in text.

The abstract from the paper is the following:

*Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlvit_architecture.jpg"
alt="drawing" width="600"/>

<small> OWL-ViT architecture. Taken from the <a href="https://arxiv.org/abs/2205.06230">original paper</a>. </small>

This model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage tips

OWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`].


key: _node_content
string_value:
{"id_": "8f8c34f7-4b77-4a94-8b17-481b3b15ab81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5ac55ae9-690b-46b5-84eb-a9228074f540", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77d1e619-da76-417e-8d4c-3c48d5c49e86", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "8c4bcb4ebe89b6a230fbd6f613aaea9a7fb9e195fca451dfc4ded5821d59ef12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f9dae157-91ea-4042-bd40-7cd5c2a214a2", "node_type": "1", "metadata": {}, "hash": "0b1eb6de101be3d103adb3b894974e3eec3d64ad5f84861710a948747eb302af", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1881, "end_char_idx": 5186, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2205.06230">original paper</a>. </small>

This model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage tips

OWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`].

```python
>>> import requests
>>> from PIL import Image
>>> import torch

>>> from transformers import OwlViTProcessor, OwlViTForObjectDetection

>>> processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
>>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> texts = [["a photo of a cat", "a photo of a dog"]]
>>> inputs = processor(text=texts, images=image, return_tensors="pt")
>>> outputs = model(**inputs)

>>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]
>>> target_sizes = torch.Tensor([image.size[::-1]])
>>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)
>>> results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)
>>> i = 0  # Retrieve predictions for the first image for the corresponding text queries
>>> text = texts[i]
>>> boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
>>> for box, score, label in zip(boxes, scores, labels):
...     box = [round(i, 2) for i in box.tolist()]
...     print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}")
Detected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29]
Detected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17]
```

## Resources

A demo notebook on using OWL-ViT for zero- and one-shot (image-guided) object detection can be found [here](https://github.com/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb).


key: _node_content
string_value:
{"id_": "f9dae157-91ea-4042-bd40-7cd5c2a214a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5ac55ae9-690b-46b5-84eb-a9228074f540", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8f8c34f7-4b77-4a94-8b17-481b3b15ab81", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "069228ac6e7f32ee56fc606c95a0475003183ae4624651071b9ce1e1f790ee8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5188, "end_char_idx": 6052, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## OwlViTConfig

[[autodoc]] OwlViTConfig
    - from_text_vision_configs

## OwlViTTextConfig

[[autodoc]] OwlViTTextConfig

## OwlViTVisionConfig

[[autodoc]] OwlViTVisionConfig

## OwlViTImageProcessor

[[autodoc]] OwlViTImageProcessor
    - preprocess
    - post_process_object_detection
    - post_process_image_guided_detection

## OwlViTFeatureExtractor

[[autodoc]] OwlViTFeatureExtractor
    - __call__
    - post_process
    - post_process_image_guided_detection

## OwlViTProcessor

[[autodoc]] OwlViTProcessor

## OwlViTModel

[[autodoc]] OwlViTModel
    - forward
    - get_text_features
    - get_image_features

## OwlViTTextModel

[[autodoc]] OwlViTTextModel
    - forward

## OwlViTVisionModel

[[autodoc]] OwlViTVisionModel
    - forward

## OwlViTForObjectDetection

[[autodoc]] OwlViTForObjectDetection
    - forward
    - image_guided_detection


key: _node_content
string_value:
{"id_": "fbeb4af5-a5ce-40f6-9b0c-a3f91c546729", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "031e2dcc-8964-4154-b4bf-3b2d0d05170e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "36f71480-52e2-4bd8-b1a5-fd3008ec1338", "node_type": "1", "metadata": {}, "hash": "f8d1190655f679eb1475dd006586be6f3ff80afc08dcd97ee8696cf225586bda", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RWKV

## Overview

The RWKV model was proposed in [this repo](https://github.com/BlinkDL/RWKV-LM)

It suggests a tweak in the traditional Transformer attention to make it linear. This way, the model can be used as recurrent network: passing inputs for timestamp 0 and timestamp 1 together is the same as passing inputs at timestamp 0, then inputs at timestamp 1 along with the state of timestamp 0 (see example below).

This can be more efficient than a regular Transformer and can deal with sentence of any length (even if the model uses a fixed context length for training).

This model was contributed by [sgugger](https://huggingface.co/sgugger).
The original code can be found [here](https://github.com/BlinkDL/RWKV-LM).

## Usage example

```py
import torch
from transformers import AutoTokenizer, RwkvConfig, RwkvModel

model = RwkvModel.from_pretrained("sgugger/rwkv-430M-pile")
tokenizer = AutoTokenizer.from_pretrained("sgugger/rwkv-430M-pile")

inputs = tokenizer("This is an example.", return_tensors="pt")
# Feed everything to the model
outputs = model(inputs["input_ids"])
output_whole = outputs.last_hidden_state

outputs = model(inputs["input_ids"][:, :2])
output_one = outputs.last_hidden_state

# Using the state computed on the first inputs, we will get the same output
outputs = model(inputs["input_ids"][:, 2:], state=outputs.state)
output_two = outputs.last_hidden_state

torch.allclose(torch.cat([output_one, output_two], dim=1), output_whole, atol=1e-5)
```

If you want to make sure the model stops generating when `'\n\n'` is detected, we recommend using the following stopping criteria:

```python 
from transformers import StoppingCriteria

class RwkvStoppingCriteria(StoppingCriteria):
    def __init__(self, eos_sequence = [187,187], eos_token_id = 537):
        self.eos_sequence = eos_sequence
        self.eos_token_id = eos_token_id

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        last_2_ids = input_ids[:,-2:].tolist()
        return self.eos_sequence in last_2_ids


output = model.generate(inputs["input_ids"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])
```

## RwkvConfig

[[autodoc]] RwkvConfig

## RwkvModel

[[autodoc]] RwkvModel
    - forward

## RwkvLMHeadModel

[[autodoc]] RwkvForCausalLM
    - forward

## Rwkv attention and the recurrent formulas

In a traditional auto-regressive Transformer, attention is written as

$$O = \hbox{softmax}(QK^{T} / \sqrt{d}) V$$

with \\(Q\\), \\(K\\) and \\(V\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\(QK^{T}\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\(V\\) to get the output \\(O\\) of the same shape as the others.


key: _node_content
string_value:
{"id_": "36f71480-52e2-4bd8-b1a5-fd3008ec1338", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "031e2dcc-8964-4154-b4bf-3b2d0d05170e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbeb4af5-a5ce-40f6-9b0c-a3f91c546729", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "735511b3960569bb826cedaa9a89fa62fbcf1af733529378c5ea4b4abc2ef438", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d34daca4-c49d-462e-bcc4-c5587ce99aaf", "node_type": "1", "metadata": {}, "hash": "45d12045c247d8d9d94d6b0070a8be57312cf7cc67a0497b44465a1bb9e6efb2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2064, "end_char_idx": 3986, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
output = model.generate(inputs["input_ids"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])
```

## RwkvConfig

[[autodoc]] RwkvConfig

## RwkvModel

[[autodoc]] RwkvModel
    - forward

## RwkvLMHeadModel

[[autodoc]] RwkvForCausalLM
    - forward

## Rwkv attention and the recurrent formulas

In a traditional auto-regressive Transformer, attention is written as

$$O = \hbox{softmax}(QK^{T} / \sqrt{d}) V$$

with \\(Q\\), \\(K\\) and \\(V\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\(QK^{T}\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\(V\\) to get the output \\(O\\) of the same shape as the others.  

Replacing the softmax by its value gives:

$$O_{i} = \frac{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}} V_{j}}{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}}}$$

Note that the entries in \\(QK^{T}\\) corresponding to \\(j > i\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).

In comparison, the RWKV attention is given by

$$O_{i} = \sigma(R_{i}) \frac{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$

where \\(R\\) is a new matrix called receptance by the author, \\(K\\) and \\(V\\) are still the key and value (\\(\sigma\\) here is the sigmoid function). \\(W\\) is a new vector that represents the position of the token and is given by

$$W_{0} = u \hbox{  and  } W_{k} = (k-1)w \hbox{ for } k \geq 1$$

with \\(u\\) and \\(w\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively.


key: _node_content
string_value:
{"id_": "d34daca4-c49d-462e-bcc4-c5587ce99aaf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "031e2dcc-8964-4154-b4bf-3b2d0d05170e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "36f71480-52e2-4bd8-b1a5-fd3008ec1338", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "8d1b70b95ef83f40dbcb4b0ce55acc38d535591b803fb44be8fb4a8c0e7b474d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d674f683-b91a-41c9-86c7-ea44c7b0810e", "node_type": "1", "metadata": {}, "hash": "9a4394f13ca6b85438cddb0baecbdb6c1cf8dcd382f551326a7abc2c410d054a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3007, "end_char_idx": 4787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Replacing the softmax by its value gives:

$$O_{i} = \frac{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}} V_{j}}{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}}}$$

Note that the entries in \\(QK^{T}\\) corresponding to \\(j > i\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).

In comparison, the RWKV attention is given by

$$O_{i} = \sigma(R_{i}) \frac{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$

where \\(R\\) is a new matrix called receptance by the author, \\(K\\) and \\(V\\) are still the key and value (\\(\sigma\\) here is the sigmoid function). \\(W\\) is a new vector that represents the position of the token and is given by

$$W_{0} = u \hbox{  and  } W_{k} = (k-1)w \hbox{ for } k \geq 1$$

with \\(u\\) and \\(w\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively. Naming them \\(N_{i}\\) and \\(D_{i}\\) we have:

$$N_{i} = e^{u + K_{i}} V_{i} + \hat{N}_{i} \hbox{  where  } \hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \cdots + e^{(i-2)w + K_{1}} V_{1}$$

so \\(\hat{N}_{i}\\) (called `numerator_state` in the code) satisfies

$$\hat{N}_{0} = 0 \hbox{  and  } \hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \hat{N}_{j}$$

and

$$D_{i} = e^{u + K_{i}} + \hat{D}_{i} \hbox{  where  } \hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \cdots + e^{(i-2)w + K_{1}}$$

so \\(\hat{D}_{i}\\) (called `denominator_state` in the code) satisfies

$$\hat{D}_{0} = 0 \hbox{  and  } \hat{D}_{j+1} = e^{K_{j}} + e^{w} \hat{D}_{j}$$

The actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers.


key: _node_content
string_value:
{"id_": "d674f683-b91a-41c9-86c7-ea44c7b0810e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "031e2dcc-8964-4154-b4bf-3b2d0d05170e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d34daca4-c49d-462e-bcc4-c5587ce99aaf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "b8f242904abfe0f180319d22a9ee88217e017df62cfbf4d87d71b5c8c3f16f9e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c16b55a-dc4d-462b-b94d-47b85de29a92", "node_type": "1", "metadata": {}, "hash": "131f8a39cbec6c5f7e458881eab01c3e4c0255e2444cc44b2aecc9c64db9c021", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3922, "end_char_idx": 5245, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The numerator and denominator can both be expressed recursively. Naming them \\(N_{i}\\) and \\(D_{i}\\) we have:

$$N_{i} = e^{u + K_{i}} V_{i} + \hat{N}_{i} \hbox{  where  } \hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \cdots + e^{(i-2)w + K_{1}} V_{1}$$

so \\(\hat{N}_{i}\\) (called `numerator_state` in the code) satisfies

$$\hat{N}_{0} = 0 \hbox{  and  } \hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \hat{N}_{j}$$

and

$$D_{i} = e^{u + K_{i}} + \hat{D}_{i} \hbox{  where  } \hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \cdots + e^{(i-2)w + K_{1}}$$

so \\(\hat{D}_{i}\\) (called `denominator_state` in the code) satisfies

$$\hat{D}_{0} = 0 \hbox{  and  } \hat{D}_{j+1} = e^{K_{j}} + e^{w} \hat{D}_{j}$$

The actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers. Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:

$$\frac{e^{x_{i}}}{\sum_{j=1}^{n} e^{x_{j}}} = \frac{e^{x_{i} - M}}{\sum_{j=1}^{n} e^{x_{j} - M}}$$

with \\(M\\) the maximum of all \\(x_{j}\\). So here on top of saving the numerator state (\\(\hat{N}\\)) and the denominator state (\\(\hat{D}\\)) we also keep track of the maximum of all terms encountered in the exponentials.


key: _node_content
string_value:
{"id_": "7c16b55a-dc4d-462b-b94d-47b85de29a92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "031e2dcc-8964-4154-b4bf-3b2d0d05170e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d674f683-b91a-41c9-86c7-ea44c7b0810e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "26ec72e9d3b6f5221ee6753093cdf86f43ad651ac91c830c60bf3a3e99d08b23", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4788, "end_char_idx": 6055, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:

$$\frac{e^{x_{i}}}{\sum_{j=1}^{n} e^{x_{j}}} = \frac{e^{x_{i} - M}}{\sum_{j=1}^{n} e^{x_{j} - M}}$$

with \\(M\\) the maximum of all \\(x_{j}\\). So here on top of saving the numerator state (\\(\hat{N}\\)) and the denominator state (\\(\hat{D}\\)) we also keep track of the maximum of all terms encountered in the exponentials. So we actually use

$$\tilde{N}_{i} = e^{-M_{i}} \hat{N}_{i} \hbox{  and  } \tilde{D}_{i} = e^{-M_{i}} \hat{D}_{i}$$

defined by the following recurrent formulas:

$$\tilde{N}_{0} = 0 \hbox{  and  } \tilde{N}_{j+1} = e^{K_{j} - q} V_{j} + e^{w + M_{j} - q} \tilde{N}_{j} \hbox{  where  } q = \max(K_{j}, w + M_{j})$$

and

$$\tilde{D}_{0} = 0 \hbox{  and  } \tilde{D}_{j+1} = e^{K_{j} - q} + e^{w + M_{j} - q} \tilde{D}_{j} \hbox{  where  } q = \max(K_{j}, w + M_{j})$$

and \\(M_{j+1} = q\\). With those, we can then compute

$$N_{i} = e^{u + K_{i} - q} V_{i} + e^{M_{i}} \tilde{N}_{i} \hbox{  where  } q = \max(u + K_{i}, M_{i})$$

and

$$D_{i} = e^{u + K_{i} - q} + e^{M_{i}} \tilde{D}_{i} \hbox{  where  } q = \max(u + K_{i}, M_{i})$$

which finally gives us

$$O_{i} = \sigma(R_{i}) \frac{N_{i}}{D_{i}}$$


key: _node_content
string_value:
{"id_": "2b42525d-9138-440b-a6b3-860b73bb091e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6a1c05cf-7a12-4603-b0d9-de66b6310f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a33d0ff1-37bb-4793-947a-9957df32631c", "node_type": "1", "metadata": {}, "hash": "d51dd502d58096dfb044b6e88d1e14bd808ca8f17971a1d98c9d9f615b8069da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DBRX

## Overview

DBRX is a [transformer-based](https://www.isattentionallyouneed.com/) decoder-only large language model (LLM) that was trained using next-token prediction.
It uses a *fine-grained* mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input.
It was pre-trained on 12T tokens of text and code data.
Compared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2.
This provides 65x more possible combinations of experts and we found that this improves model quality.
DBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA).
It is a BPE based model and uses the GPT-4 tokenizer as described in the [tiktoken](https://github.com/openai/tiktoken) repository.
We made these choices based on exhaustive evaluation and scaling experiments.

DBRX was pretrained on 12T tokens of carefully curated data and a maximum context length of 32K tokens.
We estimate that this data is at least 2x better token-for-token than the data we used to pretrain the MPT family of models.
This new dataset was developed using the full suite of Databricks tools, including Apache Spark™ and Databricks notebooks for data processing, and Unity Catalog for data management and governance.
We used curriculum learning for pretraining, changing the data mix during training in ways we found to substantially improve model quality.


More detailed information about DBRX Instruct and DBRX Base can be found in our [technical blog post](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm).

This model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.

## Usage Examples

The `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.

```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

If you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)


key: _node_content
string_value:
{"id_": "a33d0ff1-37bb-4793-947a-9957df32631c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6a1c05cf-7a12-4603-b0d9-de66b6310f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b42525d-9138-440b-a6b3-860b73bb091e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "b149284a770cf4b201410c88b19b87948d5b7cafa831b00452fb9115c5555418", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d828c850-f9ab-4597-a957-d89eac779a67", "node_type": "1", "metadata": {}, "hash": "5cb61f56f7d938e69d7b06c63e1ccdd68aac00938c7f9bf61ff3644798933570", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1768, "end_char_idx": 4634, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.

## Usage Examples

The `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.

```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

If you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="flash_attention_2",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

You can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="sdpa",
    )

input_text = "What does it take to build a great LLM?"


key: _node_content
string_value:
{"id_": "d828c850-f9ab-4597-a957-d89eac779a67", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6a1c05cf-7a12-4603-b0d9-de66b6310f00", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a33d0ff1-37bb-4793-947a-9957df32631c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "a519ffefa34499c14e1d1305f0da40d6b1e718ce83f6187dd854c212607c5772", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3068, "end_char_idx": 5086, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
(The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="flash_attention_2",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

You can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="sdpa",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

## DbrxConfig

[[autodoc]] DbrxConfig


## DbrxModel

[[autodoc]] DbrxModel
    - forward


## DbrxForCausalLM

[[autodoc]] DbrxForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "06e58cac-0e4e-4d0c-b559-418a6544eccf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7623ccc5-c515-4b8d-8db9-22b0050b8250", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5873f817-7549-4f97-82a1-d1d7f39201d8", "node_type": "1", "metadata": {}, "hash": "8e8eddd5f7ab2d7b9b24912405efe684b6db71a6036a80028ecfdfe42a89158f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ByT5

## Overview

The ByT5 model was presented in [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626) by Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir
Kale, Adam Roberts, Colin Raffel.

The abstract from the paper is the following:

*Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.
Encoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from
the model. Token-free models that instead operate directly on raw text (bytes or characters) have many benefits: they
can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by
removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token
sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of
operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with
minimal modifications to process byte sequences. We carefully characterize the trade-offs in terms of parameter count,
training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level
counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on
tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of
pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our
experiments.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://github.com/google-research/byt5).

<Tip>

ByT5's architecture is based on the T5v1.1 model, refer to [T5v1.1's documentation page](t5v1.1) for the API reference. They
only differ in how inputs should be prepared for the model, see the code examples below.

</Tip>

Since ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task
fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.


## Usage example

ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:

```python
>>> from transformers import T5ForConditionalGeneration
>>> import torch

>>> model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")

>>> num_special_tokens = 3
>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.
>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.

>>> input_ids = torch.tensor([list("Life is like a box of chocolates.".encode("utf-8"))]) + num_special_tokens

>>> labels = torch.tensor([list("La vie est comme une boîte de chocolat.


key: _node_content
string_value:
{"id_": "5873f817-7549-4f97-82a1-d1d7f39201d8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7623ccc5-c515-4b8d-8db9-22b0050b8250", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "06e58cac-0e4e-4d0c-b559-418a6544eccf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "7aa1bdd2b7ad8a226274f3e143a90d7a8458be97857aa9e1089e40d3429b0493", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "02104258-8004-475c-8edf-c835bfbd45f0", "node_type": "1", "metadata": {}, "hash": "4937b9471477d1fdadf8a629cdcdff26318a5c7b76842911025759b3e2d6bc13", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2320, "end_char_idx": 4950, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage example

ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:

```python
>>> from transformers import T5ForConditionalGeneration
>>> import torch

>>> model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")

>>> num_special_tokens = 3
>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.
>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.

>>> input_ids = torch.tensor([list("Life is like a box of chocolates.".encode("utf-8"))]) + num_special_tokens

>>> labels = torch.tensor([list("La vie est comme une boîte de chocolat.".encode("utf-8"))]) + num_special_tokens

>>> loss = model(input_ids, labels=labels).loss
>>> loss.item()
2.66
```

For batched inference and training it is however recommended to make use of the tokenizer:

```python
>>> from transformers import T5ForConditionalGeneration, AutoTokenizer

>>> model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")
>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-small")

>>> model_inputs = tokenizer(
...     ["Life is like a box of chocolates.", "Today is Monday."], padding="longest", return_tensors="pt"
... )
>>> labels_dict = tokenizer(
...     ["La vie est comme une boîte de chocolat.", "Aujourd'hui c'est lundi."], padding="longest", return_tensors="pt"
... )
>>> labels = labels_dict.input_ids

>>> loss = model(**model_inputs, labels=labels).loss
>>> loss.item()
17.9
```

Similar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, 
since the model works directly on characters, the pretraining task is a bit 
different. Let's corrupt some characters of the 
input sentence `"The dog chases a ball in the park."` and ask ByT5 to predict them 
for us.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/byt5-base")

>>> input_ids_prompt = "The dog chases a ball in the park."
>>> input_ids = tokenizer(input_ids_prompt).input_ids

>>> # Note that we cannot add "{extra_id_...}" to the string directly
>>> # as the Byte tokenizer would incorrectly merge the tokens
>>> # For ByT5, we need to work directly on the character level
>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead
>>> # uses final utf character ids.
>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.
>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.
>>> # => mask to "The dog [258]a ball [257]park."


key: _node_content
string_value:
{"id_": "02104258-8004-475c-8edf-c835bfbd45f0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7623ccc5-c515-4b8d-8db9-22b0050b8250", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5873f817-7549-4f97-82a1-d1d7f39201d8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "20100c34e76cea0c27c1ed3eef40ea016b61ed4a6e8aba5fc5225f852104060d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "563c0657-06df-498e-884c-d88de701d523", "node_type": "1", "metadata": {}, "hash": "063d4e0a601f19634d41e30172f0792ddfc1faa4844db2875f1eb8450258025d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3469, "end_char_idx": 5319, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"], padding="longest", return_tensors="pt"
... )
>>> labels_dict = tokenizer(
...     ["La vie est comme une boîte de chocolat.", "Aujourd'hui c'est lundi."], padding="longest", return_tensors="pt"
... )
>>> labels = labels_dict.input_ids

>>> loss = model(**model_inputs, labels=labels).loss
>>> loss.item()
17.9
```

Similar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, 
since the model works directly on characters, the pretraining task is a bit 
different. Let's corrupt some characters of the 
input sentence `"The dog chases a ball in the park."` and ask ByT5 to predict them 
for us.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/byt5-base")

>>> input_ids_prompt = "The dog chases a ball in the park."
>>> input_ids = tokenizer(input_ids_prompt).input_ids

>>> # Note that we cannot add "{extra_id_...}" to the string directly
>>> # as the Byte tokenizer would incorrectly merge the tokens
>>> # For ByT5, we need to work directly on the character level
>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead
>>> # uses final utf character ids.
>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.
>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.
>>> # => mask to "The dog [258]a ball [257]park."

>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])
>>> input_ids
tensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])

>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`.


key: _node_content
string_value:
{"id_": "563c0657-06df-498e-884c-d88de701d523", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7623ccc5-c515-4b8d-8db9-22b0050b8250", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "02104258-8004-475c-8edf-c835bfbd45f0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "89dac8d6a41dacc28ed77b4207dbef97a5b59faf5ed57a834743aca95780c040", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e048e8a-e9d7-40a6-bb94-b3eb02dd67c3", "node_type": "1", "metadata": {}, "hash": "16f949deb325dceb8c3100c6f57c1d74538b8b4a93110c00fd716dee66d4c944", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4046, "end_char_idx": 6522, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"` and ask ByT5 to predict them 
for us.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/byt5-base")

>>> input_ids_prompt = "The dog chases a ball in the park."
>>> input_ids = tokenizer(input_ids_prompt).input_ids

>>> # Note that we cannot add "{extra_id_...}" to the string directly
>>> # as the Byte tokenizer would incorrectly merge the tokens
>>> # For ByT5, we need to work directly on the character level
>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead
>>> # uses final utf character ids.
>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.
>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.
>>> # => mask to "The dog [258]a ball [257]park."

>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])
>>> input_ids
tensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])

>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`.
>>> output_ids = model.generate(input_ids, max_length=100)[0].tolist()
>>> output_ids
[0, 258, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118, 257,  35, 108, 113,  35, 119, 107, 104,  35, 103, 108, 118, 102, 114, 256, 108, 113,  35, 119, 107, 104, 35, 115, 100, 117, 110,  49,  35,  87, 107, 104,  35, 103, 114, 106, 35, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118,  35, 100,  35, 101, 100, 111, 111,  35, 108, 113, 255,  35, 108, 113,  35, 119, 107, 104,  35, 115, 100, 117, 110,  49]

>>> # ^- Note how 258 descends to 257, 256, 255

>>> # Now we need to split on the sentinel tokens, let's write a short loop for this
>>> output_ids_list = []
>>> start_token = 0
>>> sentinel_token = 258
>>> while sentinel_token in output_ids:
...     split_idx = output_ids.index(sentinel_token)
...     output_ids_list.append(output_ids[start_token:split_idx])
...     start_token = split_idx
...     sentinel_token -= 1

>>> output_ids_list.append(output_ids[start_token:])
>>> output_string = tokenizer.batch_decode(output_ids_list)
>>> output_string
['<pad>', 'is the one who does', ' in the disco', 'in the park.


key: _node_content
string_value:
{"id_": "2e048e8a-e9d7-40a6-bb94-b3eb02dd67c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7623ccc5-c515-4b8d-8db9-22b0050b8250", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "563c0657-06df-498e-884c-d88de701d523", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "764f4e4d4902264f1622523d98e7dbec19c3e6886060e4dfc68b7dc467ee6095", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6523, "end_char_idx": 6669, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The dog is the one who does a ball in', ' in the park.']
```


## ByT5Tokenizer

[[autodoc]] ByT5Tokenizer

See [`ByT5Tokenizer`] for all details.


key: _node_content
string_value:
{"id_": "85cf787b-c6ae-44b2-a487-622d7f062e54", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "255ca92f-35b8-41ee-9882-0bcc6c57bfad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fd8e56d-b595-4a8f-bef4-40c0998e7251", "node_type": "1", "metadata": {}, "hash": "6a9731a06952e7a5b0573498451d0fdf908e2a71fffc0d342a0e0be5c8ecf324", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SuperPoint

## Overview

The SuperPoint model was proposed
in [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629) by Daniel
DeTone, Tomasz Malisiewicz and Andrew Rabinovich.

This model is the result of a self-supervised training of a fully-convolutional network for interest point detection and
description. The model is able to detect interest points that are repeatable under homographic transformations and
provide a descriptor for each point. The use of the model in its own is limited, but it can be used as a feature
extractor for other tasks such as homography estimation, image matching, etc.

The abstract from the paper is the following:

*This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a
large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our
fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and
associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography
approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
when compared to LIFT, SIFT and ORB.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png"
alt="drawing" width="500"/>

<small> SuperPoint overview. Taken from the <a href="https://arxiv.org/abs/1712.07629v4">original paper.</a> </small>

## Usage tips

Here is a quick example of using the model to detect interest points in an image:

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
```

The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).

You can also feed multiple images to the model.


key: _node_content
string_value:
{"id_": "8fd8e56d-b595-4a8f-bef4-40c0998e7251", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "255ca92f-35b8-41ee-9882-0bcc6c57bfad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "85cf787b-c6ae-44b2-a487-622d7f062e54", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "4f6e5e9edf3cf5683a7d0d11efbc62f2bb2092fdb8c61eff3c9efcd337c49dc1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb66fd4b-48a7-4596-8cf0-b96f356d852f", "node_type": "1", "metadata": {}, "hash": "09da46a3b3df86abdc55d0ec6994a73466c32e30f9c1281aa502b712e3117a07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1100, "end_char_idx": 4342, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We introduce Homographic Adaptation, a multi-scale, multi-homography
approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
when compared to LIFT, SIFT and ORB.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png"
alt="drawing" width="500"/>

<small> SuperPoint overview. Taken from the <a href="https://arxiv.org/abs/1712.07629v4">original paper.</a> </small>

## Usage tips

Here is a quick example of using the model to detect interest points in an image:

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
```

The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).

You can also feed multiple images to the model. Due to the nature of SuperPoint, to output a dynamic number of keypoints,
you will need to use the mask attribute to retrieve the respective information :

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url_image_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
url_image_2 = "http://images.cocodataset.org/test-stuff2017/000000000568.jpg"
image_2 = Image.open(requests.get(url_image_2, stream=True).raw)

images = [image_1, image_2]

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(images, return_tensors="pt")
outputs = model(**inputs)

for i in range(len(images)):
    image_mask = outputs.mask[i]
    image_indices = torch.nonzero(image_mask).squeeze()
    image_keypoints = outputs.keypoints[i][image_indices]
    image_scores = outputs.scores[i][image_indices]
    image_descriptors = outputs.descriptors[i][image_indices]
```

You can then print the keypoints on the image to visualize the result :
```python
import cv2
for keypoint, score in zip(image_keypoints, image_scores):
    keypoint_x, keypoint_y = int(keypoint[0].item()), int(keypoint[1].item())
    color = tuple([score.item() * 255] * 3)
    image = cv2.circle(image, (keypoint_x, keypoint_y), 2, color)
cv2.imwrite("output_image.png", image)
```

This model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille).


key: _node_content
string_value:
{"id_": "bb66fd4b-48a7-4596-8cf0-b96f356d852f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "255ca92f-35b8-41ee-9882-0bcc6c57bfad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fd8e56d-b595-4a8f-bef4-40c0998e7251", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "d1a4e4198be167c360e4faf1b0b0e392f2ba722562e0c9045f0fad1f12a4c36e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4343, "end_char_idx": 5256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/magicleap/SuperPointPretrainedNetwork).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SuperPoint. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A notebook showcasing inference and visualization with SuperPoint can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SuperPoint/Inference_with_SuperPoint_to_detect_interest_points_in_an_image.ipynb). 🌎

## SuperPointConfig

[[autodoc]] SuperPointConfig

## SuperPointImageProcessor

[[autodoc]] SuperPointImageProcessor

- preprocess

## SuperPointForKeypointDetection

[[autodoc]] SuperPointForKeypointDetection

- forward


key: _node_content
string_value:
{"id_": "ac454eaa-7cc0-48b8-ab2d-ef9da77080c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c441bbb1-c28e-462b-a5e4-9a752c299f5f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "transformers"}, "hash": "052e45c889c784ad2c1f26dba252e6414d6b231f65ccd22d42dad8339a24f72b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1685, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FLAN-T5

## Overview

FLAN-T5 was released in the paper [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.

One can directly use FLAN-T5 weights without finetuning the model:

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
>>> tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")

>>> inputs = tokenizer("A step by step recipe to make bolognese pasta:", return_tensors="pt")
>>> outputs = model.generate(**inputs)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Pour a cup of bolognese into a large bowl and add the pasta']
```

FLAN-T5 includes the same improvements as T5 version 1.1 (see [here](https://huggingface.co/docs/transformers/model_doc/t5v1.1) for the full details of the model's improvements.)

Google has released the following variants:

- [google/flan-t5-small](https://huggingface.co/google/flan-t5-small)

- [google/flan-t5-base](https://huggingface.co/google/flan-t5-base)

- [google/flan-t5-large](https://huggingface.co/google/flan-t5-large)

- [google/flan-t5-xl](https://huggingface.co/google/flan-t5-xl)

- [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl).

The original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints).

<Tip>

Refer to [T5's documentation page](t5) for all API reference, code examples and notebooks. For more details regarding training and evaluation of the FLAN-T5, refer to the model card.

</Tip>


key: _node_content
string_value:
{"id_": "0f670dd5-fd6a-4971-8ecf-6b40f344cefc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c29fa2d2-ab96-4c5b-935d-2fc760425ab1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "hash": "bcb5f3c5f15a803e49e6dc0dd3250147113bbf89e466bef5c5560eb852212c90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ef2db175-bf8a-4815-8f17-b093050ea9aa", "node_type": "1", "metadata": {}, "hash": "38303f5ac0f88881c98e64927f80261931afd54ef45b4dae38fd068a8640a888", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2864, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CTRL

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=ctrl">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-ctrl-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/tiny-ctrl">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

CTRL model was proposed in [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus
of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).

The abstract from the paper is the following:

*Large-scale language models show promising text generation capabilities, but users cannot easily control particular
aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,
trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while
providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the
training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data
via model-based source attribution.*

This model was contributed by [keskarnitishr](https://huggingface.co/keskarnitishr). The original code can be found
[here](https://github.com/salesforce/ctrl).

## Usage tips

- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences
  or links to generate coherent text. Refer to the [original implementation](https://github.com/salesforce/ctrl) for
  more information.
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be
  observed in the *run_generation.py* example script.
- The PyTorch models can take the `past_key_values` as input, which is the previously computed key/value attention pairs.
  TensorFlow models accepts `past` as input. Using the `past_key_values` value prevents the model from re-computing
  pre-computed values in the context of text generation. See the [`forward`](model_doc/ctrl#transformers.CTRLModel.forward)
  method for more information on the usage of this argument.


key: _node_content
string_value:
{"id_": "ef2db175-bf8a-4815-8f17-b093050ea9aa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c29fa2d2-ab96-4c5b-935d-2fc760425ab1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "hash": "bcb5f3c5f15a803e49e6dc0dd3250147113bbf89e466bef5c5560eb852212c90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f670dd5-fd6a-4971-8ecf-6b40f344cefc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "hash": "9f1879bacff36180959022e24a0a384264c8e5f42c182c18ed7e96183a7fa3ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2867, "end_char_idx": 3600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## CTRLConfig

[[autodoc]] CTRLConfig

## CTRLTokenizer

[[autodoc]] CTRLTokenizer
    - save_vocabulary

<frameworkcontent>
<pt>

## CTRLModel

[[autodoc]] CTRLModel
    - forward

## CTRLLMHeadModel

[[autodoc]] CTRLLMHeadModel
    - forward

## CTRLForSequenceClassification

[[autodoc]] CTRLForSequenceClassification
    - forward

</pt>
<tf>

## TFCTRLModel

[[autodoc]] TFCTRLModel
    - call

## TFCTRLLMHeadModel

[[autodoc]] TFCTRLLMHeadModel
    - call

## TFCTRLForSequenceClassification

[[autodoc]] TFCTRLForSequenceClassification
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e316a2e6-ef31-4a44-a47d-32ea1d7f1218", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/git", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7aff5636-c83d-4328-ac52-71b29f18dcb0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/git", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "transformers"}, "hash": "8e695eda445ff55612b51c26331ca234eec8ab873d31219f8b7e7c098841f693", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3136, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GIT

## Overview

The GIT model was proposed in [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. GIT is a decoder-only Transformer
that leverages [CLIP](clip)'s vision encoder to condition the model on vision inputs besides text. The model obtains state-of-the-art results on
image captioning and visual question answering benchmarks.

The abstract from the paper is the following:

*In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/git_architecture.jpg"
alt="drawing" width="600"/>

<small> GIT architecture. Taken from the <a href="https://arxiv.org/abs/2205.14100" target="_blank">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).

## Usage tips

- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GIT.

- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).
- See also: [Causal language modeling task guide](../tasks/language_modeling)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.

## GitVisionConfig

[[autodoc]] GitVisionConfig

## GitVisionModel

[[autodoc]] GitVisionModel
    - forward

## GitConfig

[[autodoc]] GitConfig
    - all

## GitProcessor

[[autodoc]] GitProcessor
    - __call__

## GitModel

[[autodoc]] GitModel
    - forward

## GitForCausalLM

[[autodoc]] GitForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "a0bb4eba-6569-4f31-b93d-ba13a56d47f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clap", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e558c8cc-9b0b-4b8f-ad23-73b3ce278e44", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clap", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "transformers"}, "hash": "2952311c91ef6ca8ffcf841bfcec17524cfd1ce84e1e603f134c6e8d96ab6dce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLAP

## Overview

The CLAP model was proposed in [Large Scale Contrastive Language-Audio pretraining with
feature fusion and keyword-to-caption augmentation](https://arxiv.org/pdf/2211.06687.pdf) by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.

CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.

The abstract from the paper is the following:

*Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-6*

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .
The original code can be found [here](https://github.com/LAION-AI/Clap).

## ClapConfig

[[autodoc]] ClapConfig
    - from_text_audio_configs

## ClapTextConfig

[[autodoc]] ClapTextConfig

## ClapAudioConfig

[[autodoc]] ClapAudioConfig

## ClapFeatureExtractor

[[autodoc]] ClapFeatureExtractor

## ClapProcessor

[[autodoc]] ClapProcessor

## ClapModel

[[autodoc]] ClapModel
    - forward
    - get_text_features
    - get_audio_features

## ClapTextModel

[[autodoc]] ClapTextModel
    - forward

## ClapTextModelWithProjection

[[autodoc]] ClapTextModelWithProjection
    - forward

## ClapAudioModel

[[autodoc]] ClapAudioModel
    - forward

## ClapAudioModelWithProjection

[[autodoc]] ClapAudioModelWithProjection
    - forward


key: _node_content
string_value:
{"id_": "12616964-d157-4206-ba70-0355858e9c39", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c0e3bab-3385-44cd-a21b-a522505dbb06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "hash": "90cdafd1166bebb76ce42f5770187f53677ad7409847c087ac3f35a265a53405", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6dc79c3c-17e1-42af-b6fe-460881eb95aa", "node_type": "1", "metadata": {}, "hash": "10b1b79716168c7d477c667436e603efe4a3b6755e957eb60bc18aafe6014e1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLVP

## Overview

The CLVP (Contrastive Language-Voice Pretrained Transformer) model was proposed in [Better speech synthesis through scaling](https://arxiv.org/abs/2305.07243) by James Betker.

The abstract from the paper is the following:

*In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise - an expressive, multi-voice text-to-speech system.*


This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/neonbjb/tortoise-tts).


## Usage tips

1. CLVP is an integral part of the Tortoise TTS model.
2. CLVP can be used to compare different generated speech candidates with the provided text, and the best speech tokens are forwarded to the diffusion model.
3. The use of the [`ClvpModelForConditionalGeneration.generate()`] method is strongly recommended for tortoise usage.
4. Note that the CLVP model expects the audio to be sampled at 22.05 kHz contrary to other audio models which expects 16 kHz. 


## Brief Explanation:

- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.
- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.
- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.
- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. 
- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. 
- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  


Example :

```python
>>> import datasets
>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration

>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).
>>> text = "This is an example text."

>>> ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050))
>>> sample = ds[0]["audio"]

>>> # Define processor and model.
>>> processor = ClvpProcessor.from_pretrained("susnato/clvp_dev")
>>> model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev")

>>> # Generate processor output and model output.
>>> processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt")
>>> generated_output = model.generate(**processor_output)
```


key: _node_content
string_value:
{"id_": "6dc79c3c-17e1-42af-b6fe-460881eb95aa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2c0e3bab-3385-44cd-a21b-a522505dbb06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "hash": "90cdafd1166bebb76ce42f5770187f53677ad7409847c087ac3f35a265a53405", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12616964-d157-4206-ba70-0355858e9c39", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "hash": "0cd7f60b27916b5d3501ae71dedfd75acc6c8d6d54d515de4a8db1901fdd91a7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1436, "end_char_idx": 3958, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Brief Explanation:

- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.
- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.
- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.
- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. 
- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. 
- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  


Example :

```python
>>> import datasets
>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration

>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).
>>> text = "This is an example text."

>>> ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050))
>>> sample = ds[0]["audio"]

>>> # Define processor and model.
>>> processor = ClvpProcessor.from_pretrained("susnato/clvp_dev")
>>> model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev")

>>> # Generate processor output and model output.
>>> processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt")
>>> generated_output = model.generate(**processor_output)
```


## ClvpConfig

[[autodoc]] ClvpConfig
    - from_sub_model_configs

## ClvpEncoderConfig

[[autodoc]] ClvpEncoderConfig

## ClvpDecoderConfig

[[autodoc]] ClvpDecoderConfig

## ClvpTokenizer

[[autodoc]] ClvpTokenizer
    - save_vocabulary

## ClvpFeatureExtractor

[[autodoc]] ClvpFeatureExtractor
    - __call__

## ClvpProcessor

[[autodoc]] ClvpProcessor
    - __call__
    - decode
    - batch_decode

## ClvpModelForConditionalGeneration

[[autodoc]] ClvpModelForConditionalGeneration
    - forward
    - generate
    - get_text_features
    - get_speech_features

## ClvpForCausalLM

[[autodoc]] ClvpForCausalLM

## ClvpModel

[[autodoc]] ClvpModel

## ClvpEncoder

[[autodoc]] ClvpEncoder

## ClvpDecoder

[[autodoc]] ClvpDecoder


key: _node_content
string_value:
{"id_": "85e4ab77-de87-4746-b8a8-52e8030b24c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7ebd18f5-e7eb-42ca-a55f-3b1e37f877b4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "transformers"}, "hash": "baf5f71b6e8653a8aca416eef2e7a45d85b5901d8481ce2cadf41070e00d7585", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RecurrentGemma

## Overview

The Recurrent Gemma model was proposed in [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://storage.googleapis.com/deepmind-media/gemma/recurrentgemma-report.pdf) by the Griffin, RLHF and Gemma Teams of Google.

The abstract from the paper is the following:

*We introduce RecurrentGemma, an open language model which uses Google’s novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.*

Tips:

- The original checkpoints can be converted using the conversion script [`src/transformers/models/recurrent_gemma/convert_recurrent_gemma_weights_to_hf.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/recurrent_gemma/convert_recurrent_gemma_to_hf.py). 

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The original code can be found [here](https://github.com/google-deepmind/recurrentgemma).


## RecurrentGemmaConfig

[[autodoc]] RecurrentGemmaConfig


## RecurrentGemmaModel

[[autodoc]] RecurrentGemmaModel
    - forward

## RecurrentGemmaForCausalLM

[[autodoc]] RecurrentGemmaForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "24ad6a3e-3802-4181-acb5-b271e45b9b5f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "05badb53-4862-4ad5-aa0c-3203e1a69dcb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25df1612-c9e4-4b6b-8c2e-9f96683f62d4", "node_type": "1", "metadata": {}, "hash": "7cc0bb5b80177fc43726039da8bbe670c7e2fc7261cb0c4965eaced23a0ba463", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3289, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DETR

## Overview

The DETR model was proposed in [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. DETR
consists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for
object detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use
things like region proposals, non-maximum suppression procedure and anchor generation. Moreover, DETR can also be
naturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.

The abstract from the paper is the following:

*We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the
detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression
procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the
new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via
bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries,
DETR reasons about the relations of the objects and the global image context to directly output the final set of
predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many
other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and
highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily
generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive
baselines.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).

## How DETR works

Here's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:

First, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use
ResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a
tensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone
outputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is
then projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a
`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the
feature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =
`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually
longer than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).

Next, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider
these as image features).


key: _node_content
string_value:
{"id_": "25df1612-c9e4-4b6b-8c2e-9f96683f62d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "05badb53-4862-4ad5-aa0c-3203e1a69dcb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24ad6a3e-3802-4181-acb5-b271e45b9b5f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "8133511a188276b66d7ff74afe79471446075eaf74533ef3921005f4f451f77c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c396e610-1971-4c78-bbec-ce5b44ed130f", "node_type": "1", "metadata": {}, "hash": "23941821e25ce2b1aa060ae85a540c4ab9aca2358db261fdbc09e5e92424a9fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1778, "end_char_idx": 5025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


