# embedding_metadata

key: chroma:document
string_value:
You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import torch
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


>>> class TableDataset(torch.utils.data.Dataset):
...     def __init__(self, data, tokenizer):
...         self.data = data
...         self.tokenizer = tokenizer

...     def __getitem__(self, idx):
...         item = data.iloc[idx]
...         table = pd.read_csv(table_csv_path + item.table_file).astype(
...             str
...         )  # be sure to make your table data text only
...         encoding = self.tokenizer(
...             table=table,
...             queries=item.question,
...             answer_coordinates=item.answer_coordinates,
...             answer_text=item.answer_text,
...             truncation=True,
...             padding="max_length",
...             return_tensors="pt",
...         )
...         # remove the batch dimension which the tokenizer adds by default
...         encoding = {key: val.squeeze(0) for key, val in encoding.items()}
...         # add the float_answer which is also required (weak supervision for aggregation case)
...         encoding["float_answer"] = torch.tensor(item.float_answer)
...         return encoding

...     def __len__(self):
...         return len(self.data)


>>> data = pd.read_csv(tsv_path, sep="\t")
>>> train_dataset = TableDataset(data, tokenizer)
>>> train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)
```
</pt>
<tf>
Third, given that you've prepared your data in this TSV/CSV format (and corresponding CSV files containing the tabular data), you can then use [`TapasTokenizer`] to convert table-question pairs into `input_ids`, `attention_mask`, `token_type_ids` and so on. Again, based on which of the three cases you picked above, [`TFTapasForQuestionAnswering`] requires different
inputs to be fine-tuned:

| **Task**                           | **Required inputs**                                                                                                 |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| Conversational                     | `input_ids`, `attention_mask`, `token_type_ids`, `labels`                                                           |
|  Weak supervision for aggregation  | `input_ids`, `attention_mask`, `token_type_ids`, `labels`, `numeric_values`, `numeric_values_scale`, `float_answer` |
| Strong supervision for aggregation | `input ids`, `attention mask`, `token type ids`, `labels`, `aggregation_labels`                                     |

[`TapasTokenizer`] creates the `labels`, `numeric_values` and `numeric_values_scale` based on the `answer_coordinates` and `answer_text` columns of the TSV file. The `float_answer` and `aggregation_labels` are already in the TSV file of step 2. Here's an example:

```py
>>> from transformers import TapasTokenizer
>>> import pandas as pd

>>> model_name = "google/tapas-base"
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> answer_coordinates = [[(0, 0)], [(2, 1)], [(0, 1), (1, 1), (2, 1)]]
>>> answer_text = [["Brad Pitt"], ["69"], ["209"]]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(
...     table=table,
...     queries=queries,
...     answer_coordinates=answer_coordinates,
...     answer_text=answer_text,
...     padding="max_length",
...     return_tensors="tf",
... )
>>> inputs
{'input_ids': tensor([[ ... ]]), 'attention_mask': tensor([[...]]), 'token_type_ids': tensor([[[...]]]),
'numeric_values': tensor([[ ... ]]), 'numeric_values_scale: tensor([[ ... ]]), labels: tensor([[ ... ]])}
```

Note that [`TapasTokenizer`] expects the data of the table to be **text-only**. You can use `.astype(str)` on a dataframe to turn it into text-only data.
Of course, this only shows how to encode a single training example. It is advised to create a dataloader to iterate over batches:

```py
>>> import tensorflow as tf
>>> import pandas as pd

>>> tsv_path = "your_path_to_the_tsv_file"
>>> table_csv_path = "your_path_to_a_directory_containing_all_csv_files"


key: _node_content
string_value:
{"id_": "af66a608-9b0e-49b6-a33c-53304ba83a6c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05d6bbe7-3387-4c08-be1c-ed57625792d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "d3bef3c03a0bb62fdfd22809525e3b40d9573bf3c045bc3fcc165f1da3ad634f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a", "node_type": "1", "metadata": {}, "hash": "bf3e72011ba82d8e9164c074330aba58095415c84d7dd5ec0f3f6aa5c8b619c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> class TableDataset:
...     def __init__(self, data, tokenizer):
...         self.data = data
...         self.tokenizer = tokenizer

...     def __iter__(self):
...         for idx in range(self.__len__()):
...             item = self.data.iloc[idx]
...             table = pd.read_csv(table_csv_path + item.table_file).astype(
...                 str
...             )  # be sure to make your table data text only
...             encoding = self.tokenizer(
...                 table=table,
...                 queries=item.question,
...                 answer_coordinates=item.answer_coordinates,
...                 answer_text=item.answer_text,
...                 truncation=True,
...                 padding="max_length",
...                 return_tensors="tf",
...             )
...             # remove the batch dimension which the tokenizer adds by default
...             encoding = {key: tf.squeeze(val, 0) for key, val in encoding.items()}
...             # add the float_answer which is also required (weak supervision for aggregation case)
...             encoding["float_answer"] = tf.convert_to_tensor(item.float_answer, dtype=tf.float32)
...             yield encoding["input_ids"], encoding["attention_mask"], encoding["numeric_values"], encoding[
...                 "numeric_values_scale"
...             ], encoding["token_type_ids"], encoding["labels"], encoding["float_answer"]

...     def __len__(self):
...         return len(self.data)


>>> data = pd.read_csv(tsv_path, sep="\t")
>>> train_dataset = TableDataset(data, tokenizer)
>>> output_signature = (
...     tf.TensorSpec(shape=(512,), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.float32),
...     tf.TensorSpec(shape=(512,), dtype=tf.float32),
...     tf.TensorSpec(shape=(512, 7), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.int32),
...     tf.TensorSpec(shape=(512,), dtype=tf.float32),
... )
>>> train_dataloader = tf.data.Dataset.from_generator(train_dataset, output_signature=output_signature).batch(32)
```
</tf>
</frameworkcontent>

Note that here, we encode each table-question pair independently. This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`
index) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.

**STEP 4: Train (fine-tune) the model

<frameworkcontent>
<pt>
You can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.


key: _node_content
string_value:
{"id_": "4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "af66a608-9b0e-49b6-a33c-53304ba83a6c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "9287cc4bb0f5f09d33e75fe4210174a0942bac5a08a0ea1aae8e3dbcd6613642", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b1188f1-3371-46d7-9fe5-c31ce529bad4", "node_type": "1", "metadata": {}, "hash": "473e2b92ae58897c38b18d4a0d30fd273ca0a21b89f9be891ebcc87d865bcc4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is fine as long as your dataset is **not conversational**. In case your dataset involves conversational questions (such as in SQA), then you should first group together the `queries`, `answer_coordinates` and `answer_text` per table (in the order of their `position`
index) and batch encode each table with its questions. This will make sure that the `prev_labels` token types (see docs of [`TapasTokenizer`]) are set correctly. See [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info. See [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) for more info regarding using the TensorFlow model.

**STEP 4: Train (fine-tune) the model

<frameworkcontent>
<pt>
You can then fine-tune [`TapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> from transformers import TapasConfig, TapasForQuestionAnswering, AdamW

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.     init_cell_selection_weights_to_zero=True,
.     select_one_column=True,
.     allow_empty_column_selection=False,
.     temperature=0.0352513,
. )
>>> model = TapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> optimizer = AdamW(model.parameters(), lr=5e-5)

>>> model.train()
>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch["input_ids"]
.         attention_mask = batch["attention_mask"]
.         token_type_ids = batch["token_type_ids"]
.         labels = batch["labels"]
.         numeric_values = batch["numeric_values"]
.         numeric_values_scale = batch["numeric_values_scale"]
.         float_answer = batch["float_answer"]

.         # zero the parameter gradients
.         optimizer.zero_grad()

.         # forward + backward + optimize
.         outputs = model(
.             input_ids=input_ids,
.             attention_mask=attention_mask,
.             token_type_ids=token_type_ids,
.             labels=labels,
.             numeric_values=numeric_values,
.             numeric_values_scale=numeric_values_scale,
.             float_answer=float_answer,
.         )
.         loss = outputs.loss
.         loss.backward()
.         optimizer.step()
```
</pt>
<tf>
You can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> import tensorflow as tf
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.


key: _node_content
string_value:
{"id_": "4b1188f1-3371-46d7-9fe5-c31ce529bad4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b1a7dbb-4ac7-4b23-b0fc-95fa6269719a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "79d4a4602f87a5f2395579edb72aaf4fc3b8c112d896e5a6c9d27da941f0874c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4a47dd7-b667-401d-a3ea-259b68e75347", "node_type": "1", "metadata": {}, "hash": "13daf7eb36d0d077b78ee3959811fe83697d4b661e94c8eca429d1b6b7bb72f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
from_pretrained("google/tapas-base", config=config)

>>> optimizer = AdamW(model.parameters(), lr=5e-5)

>>> model.train()
>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch["input_ids"]
.         attention_mask = batch["attention_mask"]
.         token_type_ids = batch["token_type_ids"]
.         labels = batch["labels"]
.         numeric_values = batch["numeric_values"]
.         numeric_values_scale = batch["numeric_values_scale"]
.         float_answer = batch["float_answer"]

.         # zero the parameter gradients
.         optimizer.zero_grad()

.         # forward + backward + optimize
.         outputs = model(
.             input_ids=input_ids,
.             attention_mask=attention_mask,
.             token_type_ids=token_type_ids,
.             labels=labels,
.             numeric_values=numeric_values,
.             numeric_values_scale=numeric_values_scale,
.             float_answer=float_answer,
.         )
.         loss = outputs.loss
.         loss.backward()
.         optimizer.step()
```
</pt>
<tf>
You can then fine-tune [`TFTapasForQuestionAnswering`] as follows (shown here for the weak supervision for aggregation case):

```py
>>> import tensorflow as tf
>>> from transformers import TapasConfig, TFTapasForQuestionAnswering

>>> # this is the default WTQ configuration
>>> config = TapasConfig(
.     num_aggregation_labels=4,
.     use_answer_as_supervision=True,
.     answer_loss_cutoff=0.664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.     init_cell_selection_weights_to_zero=True,
.     select_one_column=True,
.     allow_empty_column_selection=False,
.     temperature=0.0352513,
. )
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)

>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch[0]
.         attention_mask = batch[1]
.         token_type_ids = batch[4]
.         labels = batch[-1]
.         numeric_values = batch[2]
.         numeric_values_scale = batch[3]
.         float_answer = batch[6]

.         # forward + backward + optimize
.         with tf.GradientTape() as tape:
.             outputs = model(
.                 input_ids=input_ids,
.                 attention_mask=attention_mask,
.                 token_type_ids=token_type_ids,
.                 labels=labels,
.                 numeric_values=numeric_values,
.                 numeric_values_scale=numeric_values_scale,
.                 float_answer=float_answer,
.             )
.         grads = tape.gradient(outputs.loss, model.trainable_weights)
.         optimizer.apply_gradients(zip(grads, model.trainable_weights))
```
</tf>
</frameworkcontent>

## Usage: inference

<frameworkcontent>
<pt>
Here we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data).


key: _node_content
string_value:
{"id_": "c4a47dd7-b667-401d-a3ea-259b68e75347", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b1188f1-3371-46d7-9fe5-c31ce529bad4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "2c3c642019f4dea310311c0de08ffd38fc99049d28518bfc687bd6adfeb6e420", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1017f0df-6bca-497d-a4aa-808c218a0dc5", "node_type": "1", "metadata": {}, "hash": "89dd0ff4dff0907c673ed19305715fa40b1bf3be6c9bccde79f40b78654ec9ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
664694,
.     cell_selection_preference=0.207951,
.     huber_loss_delta=0.121194,
.     init_cell_selection_weights_to_zero=True,
.     select_one_column=True,
.     allow_empty_column_selection=False,
.     temperature=0.0352513,
. )
>>> model = TFTapasForQuestionAnswering.from_pretrained("google/tapas-base", config=config)

>>> optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)

>>> for epoch in range(2):  # loop over the dataset multiple times
.     for batch in train_dataloader:
.         # get the inputs;
.         input_ids = batch[0]
.         attention_mask = batch[1]
.         token_type_ids = batch[4]
.         labels = batch[-1]
.         numeric_values = batch[2]
.         numeric_values_scale = batch[3]
.         float_answer = batch[6]

.         # forward + backward + optimize
.         with tf.GradientTape() as tape:
.             outputs = model(
.                 input_ids=input_ids,
.                 attention_mask=attention_mask,
.                 token_type_ids=token_type_ids,
.                 labels=labels,
.                 numeric_values=numeric_values,
.                 numeric_values_scale=numeric_values_scale,
.                 float_answer=float_answer,
.             )
.         grads = tape.gradient(outputs.loss, model.trainable_weights)
.         optimizer.apply_gradients(zip(grads, model.trainable_weights))
```
</tf>
</frameworkcontent>

## Usage: inference

<frameworkcontent>
<pt>
Here we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?


key: _node_content
string_value:
{"id_": "1017f0df-6bca-497d-a4aa-808c218a0dc5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4a47dd7-b667-401d-a3ea-259b68e75347", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "70269f57b62152b6fcb733717729e470bf8f4bfd2b3a0fd735a9eb19ad7b7b96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa70dce4-b195-42c7-a324-102ff2903344", "node_type": "1", "metadata": {}, "hash": "9717b0fabfb02fcb3c2ead98d633de276ed56bd1eab76e6e02ce5d15ce051801", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
numeric_values=numeric_values,
.                 numeric_values_scale=numeric_values_scale,
.                 float_answer=float_answer,
.             )
.         grads = tape.gradient(outputs.loss, model.trainable_weights)
.         optimizer.apply_gradients(zip(grads, model.trainable_weights))
```
</tf>
</frameworkcontent>

## Usage: inference

<frameworkcontent>
<pt>
Here we explain how you can use [`TapasForQuestionAnswering`] or [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> outputs = model(**inputs)
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()
... )

>>> # let's print out the results:
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3: "COUNT"}
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]

>>> answers = []
>>> for coordinates in predicted_answer_coordinates:
...     if len(coordinates) == 1:
...         # only a single cell:
...         answers.append(table.iat[coordinates[0]])
...     else:
...         # multiple cells
...         cell_values = []
...         for coordinate in coordinates:
...             cell_values.append(table.iat[coordinate])
...         answers.append(", ".join(cell_values))

>>> display(table)
>>> print("")
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
...     print(query)
...     if predicted_agg == "NONE":
...         print("Predicted answer: " + answer)
...     else:
...         print("Predicted answer: " + predicted_agg + " > " + answer)
What is the name of the first actor?
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</pt>
<tf>
Here we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e.


key: _node_content
string_value:
{"id_": "fa70dce4-b195-42c7-a324-102ff2903344", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1017f0df-6bca-497d-a4aa-808c218a0dc5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "20c2c399cd2ed793df9058b5015b196735f266b6d18c9b7ce1152483cc6d226e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c30f659-7919-41e1-b5ee-62c659a19dd3", "node_type": "1", "metadata": {}, "hash": "16148826af60d7d6f5f0365c8155bc86b92cb18b7174ca5fb72da00db3919f50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30325, "end_char_idx": 33178, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     "What is the total number of movies?",
... ]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="pt")
>>> outputs = model(**inputs)
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
...     inputs, outputs.logits.detach(), outputs.logits_aggregation.detach()
... )

>>> # let's print out the results:
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3: "COUNT"}
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]

>>> answers = []
>>> for coordinates in predicted_answer_coordinates:
...     if len(coordinates) == 1:
...         # only a single cell:
...         answers.append(table.iat[coordinates[0]])
...     else:
...         # multiple cells
...         cell_values = []
...         for coordinate in coordinates:
...             cell_values.append(table.iat[coordinate])
...         answers.append(", ".join(cell_values))

>>> display(table)
>>> print("")
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
...     print(query)
...     if predicted_agg == "NONE":
...         print("Predicted answer: " + answer)
...     else:
...         print("Predicted answer: " + predicted_agg + " > " + answer)
What is the name of the first actor?
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</pt>
<tf>
Here we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?


key: _node_content
string_value:
{"id_": "7c30f659-7919-41e1-b5ee-62c659a19dd3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa70dce4-b195-42c7-a324-102ff2903344", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "42905cb43de7c5c6c7d27340b15d198b991a17a4145339957edfc35cbec2dc2b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb9bd799-defe-4865-80b0-f8f9a508b01c", "node_type": "1", "metadata": {}, "hash": "22475e8c44e5f9d42ac182183fa5e68edfe42edfaed79b206d56d8a84cf538bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31715, "end_char_idx": 34943, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</pt>
<tf>
Here we explain how you can use [`TFTapasForQuestionAnswering`] for inference (i.e. making predictions on new data). For inference, only `input_ids`, `attention_mask` and `token_type_ids` (which you can obtain using [`TapasTokenizer`]) have to be provided to the model to obtain the logits. Next, you can use the handy [`~models.tapas.tokenization_tapas.convert_logits_to_predictions`] method to convert these into predicted coordinates and optional aggregation indices.

However, note that inference is **different** depending on whether or not the setup is conversational. In a non-conversational set-up, inference can be done in parallel on all table-question pairs of a batch. Here's an example of that:

```py
>>> from transformers import TapasTokenizer, TFTapasForQuestionAnswering
>>> import pandas as pd

>>> model_name = "google/tapas-base-finetuned-wtq"
>>> model = TFTapasForQuestionAnswering.from_pretrained(model_name)
>>> tokenizer = TapasTokenizer.from_pretrained(model_name)

>>> data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
>>> queries = [
...     "What is the name of the first actor?",
...     "How many movies has George Clooney played in?",
...     "What is the total number of movies?",
... ]
>>> table = pd.DataFrame.from_dict(data)
>>> inputs = tokenizer(table=table, queries=queries, padding="max_length", return_tensors="tf")
>>> outputs = model(**inputs)
>>> predicted_answer_coordinates, predicted_aggregation_indices = tokenizer.convert_logits_to_predictions(
...     inputs, outputs.logits, outputs.logits_aggregation
... )

>>> # let's print out the results:
>>> id2aggregation = {0: "NONE", 1: "SUM", 2: "AVERAGE", 3: "COUNT"}
>>> aggregation_predictions_string = [id2aggregation[x] for x in predicted_aggregation_indices]

>>> answers = []
>>> for coordinates in predicted_answer_coordinates:
...     if len(coordinates) == 1:
...         # only a single cell:
...         answers.append(table.iat[coordinates[0]])
...     else:
...         # multiple cells
...         cell_values = []
...         for coordinate in coordinates:
...             cell_values.append(table.iat[coordinate])
...         answers.append(", ".join(cell_values))

>>> display(table)
>>> print("")
>>> for query, answer, predicted_agg in zip(queries, answers, aggregation_predictions_string):
...     print(query)
...     if predicted_agg == "NONE":
...         print("Predicted answer: " + answer)
...     else:
...         print("Predicted answer: " + predicted_agg + " > " + answer)
What is the name of the first actor?
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</tf>
</frameworkcontent>

In case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair.


key: _node_content
string_value:
{"id_": "fb9bd799-defe-4865-80b0-f8f9a508b01c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f9af8e86-dc56-45f2-b03e-4c0dec92cf4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "8e6130f27a1afbb461954b6f5888d6a84337bc9f7cc137ee5faef9c4015e0421", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c30f659-7919-41e1-b5ee-62c659a19dd3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tapas", "title": "TAPAS", "tokens": 8355, "retrieve_doc": false, "source": "transformers"}, "hash": "e58991972fc6735dfa3b608f3b432ebba2d704a9573b469d180f1eebec0481d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34503, "end_char_idx": 36359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Predicted answer: Brad Pitt
How many movies has George Clooney played in?
Predicted answer: COUNT > 69
What is the total number of movies?
Predicted answer: SUM > 87, 53, 69
```
</tf>
</frameworkcontent>

In case of a conversational set-up, then each table-question pair must be provided **sequentially** to the model, such that the `prev_labels` token types can be overwritten by the predicted `labels` of the previous table-question pair. Again, more info can be found in [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for PyTorch) and [this notebook](https://github.com/kamalkraj/Tapas-Tutorial/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) (for TensorFlow).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Masked language modeling task guide](../tasks/masked_language_modeling)

## TAPAS specific outputs
[[autodoc]] models.tapas.modeling_tapas.TableQuestionAnsweringOutput

## TapasConfig
[[autodoc]] TapasConfig

## TapasTokenizer
[[autodoc]] TapasTokenizer
    - __call__
    - convert_logits_to_predictions
    - save_vocabulary

<frameworkcontent>
<pt>

## TapasModel
[[autodoc]] TapasModel
    - forward
    
## TapasForMaskedLM
[[autodoc]] TapasForMaskedLM
    - forward

## TapasForSequenceClassification
[[autodoc]] TapasForSequenceClassification
    - forward
    
## TapasForQuestionAnswering
[[autodoc]] TapasForQuestionAnswering
    - forward

</pt>
<tf>

## TFTapasModel
[[autodoc]] TFTapasModel
    - call
    
## TFTapasForMaskedLM
[[autodoc]] TFTapasForMaskedLM
    - call

## TFTapasForSequenceClassification
[[autodoc]] TFTapasForSequenceClassification
    - call
    
## TFTapasForQuestionAnswering
[[autodoc]] TFTapasForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "34b34841-00a8-42e8-8672-9124f64e3b5a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "hash": "824354140f99d988dc7a66f6a4824d0281e10ec39c4c3654187d84104ce96d51", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "314e5d4e-53fe-43e9-b0f1-d4c30ca5752d", "node_type": "1", "metadata": {}, "hash": "e564a0391f04e2e1af5b3fcec19d1145d103f78db3cebc46031ee9100fda3cb6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3312, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LXMERT

## Overview

The LXMERT model was proposed in [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://arxiv.org/abs/1908.07490) by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders
(one for the vision modality, one for the language modality, and then one to fuse both modalities) pretrained using a
combination of masked language modeling, visual-language text alignment, ROI-feature regression, masked
visual-attribute modeling, masked visual-object modeling, and visual-question answering objectives. The pretraining
consists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering, VQA 2.0, and GQA.

The abstract from the paper is the following:

*Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly,
the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality
Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we
build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language
encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language
semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative
pretraining tasks: masked language modeling, masked object prediction (feature regression and label classification),
cross-modality matching, and image question answering. These tasks help in learning both intra-modality and
cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art
results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our
pretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous
best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel
model components and pretraining strategies significantly contribute to our strong results; and also present several
attention visualizations for the different encoders*

This model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).

## Usage tips

- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features
  will work.
- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the
  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to
  itself, select the vision/language hidden states from the first input in the tuple.
- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used
  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder
  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and
  both self attention outputs are disregarded.


key: _node_content
string_value:
{"id_": "314e5d4e-53fe-43e9-b0f1-d4c30ca5752d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "893e5b19-96fc-49d7-8d12-1ab97b8bf67c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "hash": "824354140f99d988dc7a66f6a4824d0281e10ec39c4c3654187d84104ce96d51", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "34b34841-00a8-42e8-8672-9124f64e3b5a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lxmert", "title": "LXMERT", "tokens": 1183, "retrieve_doc": true, "source": "transformers"}, "hash": "a80a91485425c84cebe9bbc35b31447b0cf52cdaaad664256041bce0637452ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1618, "end_char_idx": 4322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These tasks help in learning both intra-modality and
cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art
results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our
pretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous
best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel
model components and pretraining strategies significantly contribute to our strong results; and also present several
attention visualizations for the different encoders*

This model was contributed by [eltoto1219](https://huggingface.co/eltoto1219). The original code can be found [here](https://github.com/airsplay/lxmert).

## Usage tips

- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features
  will work.
- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the
  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to
  itself, select the vision/language hidden states from the first input in the tuple.
- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used
  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder
  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and
  both self attention outputs are disregarded.

## Resources

- [Question answering task guide](../tasks/question_answering)

## LxmertConfig

[[autodoc]] LxmertConfig

## LxmertTokenizer

[[autodoc]] LxmertTokenizer

## LxmertTokenizerFast

[[autodoc]] LxmertTokenizerFast

## Lxmert specific outputs

[[autodoc]] models.lxmert.modeling_lxmert.LxmertModelOutput

[[autodoc]] models.lxmert.modeling_lxmert.LxmertForPreTrainingOutput

[[autodoc]] models.lxmert.modeling_lxmert.LxmertForQuestionAnsweringOutput

[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertModelOutput

[[autodoc]] models.lxmert.modeling_tf_lxmert.TFLxmertForPreTrainingOutput

<frameworkcontent>
<pt>

## LxmertModel

[[autodoc]] LxmertModel
    - forward

## LxmertForPreTraining

[[autodoc]] LxmertForPreTraining
    - forward

## LxmertForQuestionAnswering

[[autodoc]] LxmertForQuestionAnswering
    - forward

</pt>
<tf>

## TFLxmertModel

[[autodoc]] TFLxmertModel
    - call

## TFLxmertForPreTraining

[[autodoc]] TFLxmertForPreTraining
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "9b0457b7-6ebe-4dfc-9bb0-d0719b65a039", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "hash": "17fd551fd712ca727ea3d1a97dc6e44926a8ce1ac6daf07d80f77850878fa9f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2db6aab4-fd2c-4146-8fcc-d0dff2738b24", "node_type": "1", "metadata": {}, "hash": "ef621a0e76d8e3fd4223cefff8da8a75cce207dd7f7212953fc6e8e55d23a5c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3416, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DPT

## Overview

The DPT model was proposed in [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by René Ranftl, Alexey Bochkovskiy, Vladlen Koltun.
DPT is a model that leverages the [Vision Transformer (ViT)](vit) as backbone for dense prediction tasks like semantic segmentation and depth estimation.

The abstract from the paper is the following:

*We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense vision transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg"
alt="drawing" width="600"/>

<small> DPT architecture. Taken from the <a href="https://arxiv.org/abs/2103.13413" target="_blank">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).

## Usage tips

DPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:

```python
from transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation

# initialize with a Transformer-based backbone such as DINOv2
# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)
backbone_config = Dinov2Config.from_pretrained("facebook/dinov2-base", out_features=["stage1", "stage2", "stage3", "stage4"], reshape_hidden_states=False)

config = DPTConfig(backbone_config=backbone_config)
model = DPTForDepthEstimation(config=config)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DPT.

- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).

- [Semantic segmentation task guide](../tasks/semantic_segmentation)
- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!


key: _node_content
string_value:
{"id_": "2db6aab4-fd2c-4146-8fcc-d0dff2738b24", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3515131e-ba54-4f46-8fd5-484448e3e393", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "hash": "17fd551fd712ca727ea3d1a97dc6e44926a8ce1ac6daf07d80f77850878fa9f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b0457b7-6ebe-4dfc-9bb0-d0719b65a039", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dpt", "title": "DPT", "tokens": 1085, "retrieve_doc": true, "source": "transformers"}, "hash": "70a20fe5895b116c67d4d36ee3bfe27be499f9c990e4b9bef75cc8c8f33fc4b9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1941, "end_char_idx": 3985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/isl-org/DPT).

## Usage tips

DPT is compatible with the [`AutoBackbone`] class. This allows to use the DPT framework with various computer vision backbones available in the library, such as [`VitDetBackbone`] or [`Dinov2Backbone`]. One can create it as follows:

```python
from transformers import Dinov2Config, DPTConfig, DPTForDepthEstimation

# initialize with a Transformer-based backbone such as DINOv2
# in that case, we also specify `reshape_hidden_states=False` to get feature maps of shape (batch_size, num_channels, height, width)
backbone_config = Dinov2Config.from_pretrained("facebook/dinov2-base", out_features=["stage1", "stage2", "stage3", "stage4"], reshape_hidden_states=False)

config = DPTConfig(backbone_config=backbone_config)
model = DPTForDepthEstimation(config=config)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DPT.

- Demo notebooks for [`DPTForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DPT).

- [Semantic segmentation task guide](../tasks/semantic_segmentation)
- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DPTConfig

[[autodoc]] DPTConfig

## DPTFeatureExtractor

[[autodoc]] DPTFeatureExtractor
    - __call__
    - post_process_semantic_segmentation

## DPTImageProcessor

[[autodoc]] DPTImageProcessor
    - preprocess
    - post_process_semantic_segmentation

## DPTModel

[[autodoc]] DPTModel
    - forward

## DPTForDepthEstimation

[[autodoc]] DPTForDepthEstimation
    - forward

## DPTForSemanticSegmentation

[[autodoc]] DPTForSemanticSegmentation
    - forward


key: _node_content
string_value:
{"id_": "8b64bcdc-f76c-4c8b-8c23-cda43fadc390", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "hash": "f59dcff97f3e1d243275ecb5e64ce123396e3adbfb60ef6e63c92dba3da20e78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f91954f6-1c59-463d-950d-56ac2644a817", "node_type": "1", "metadata": {}, "hash": "b1d5a869de85b830a05286ec1e0ae4567c892fefbab81eece9fb43d3f749df7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3319, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CANINE

## Overview

The CANINE model was proposed in [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language
Representation](https://arxiv.org/abs/2103.06874) by Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting. It's
among the first papers that trains a Transformer without using an explicit tokenization step (such as Byte Pair
Encoding (BPE), WordPiece or SentencePiece). Instead, the model is trained directly at a Unicode character-level.
Training at a character-level inevitably comes with a longer sequence length, which CANINE solves with an efficient
downsampling strategy, before applying a deep Transformer encoder.

The abstract from the paper is the following:

*Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models
still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword
lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all
languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE,
a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a
pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias.
To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input
sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by
2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/google-research/language/tree/master/language/canine).

## Usage tips

- CANINE uses no less than 3 Transformer encoders internally: 2 "shallow" encoders (which only consist of a single
  layer) and 1 "deep" encoder (which is a regular BERT encoder). First, a "shallow" encoder is used to contextualize
  the character embeddings, using local attention. Next, after downsampling, a "deep" encoder is applied. Finally,
  after upsampling, a "shallow" encoder is used to create the final character embeddings. Details regarding up- and
  downsampling can be found in the paper.
- CANINE uses a max sequence length of 2048 characters by default. One can use [`CanineTokenizer`]
  to prepare text for the model.
- Classification can be done by placing a linear layer on top of the final hidden state of the special [CLS] token
  (which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of
  tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The
  details for this can be found in the paper.

Model checkpoints:

  - [google/canine-c](https://huggingface.co/google/canine-c): Pre-trained with autoregressive character loss,
    12-layer, 768-hidden, 12-heads, 121M parameters (size ~500 MB).
  - [google/canine-s](https://huggingface.co/google/canine-s): Pre-trained with subword loss, 12-layer,
    768-hidden, 12-heads, 121M parameters (size ~500 MB).


key: _node_content
string_value:
{"id_": "f91954f6-1c59-463d-950d-56ac2644a817", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "57fc9403-20ca-479d-a89e-7644b11f66bf", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "hash": "f59dcff97f3e1d243275ecb5e64ce123396e3adbfb60ef6e63c92dba3da20e78", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b64bcdc-f76c-4c8b-8c23-cda43fadc390", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/canine", "title": "CANINE", "tokens": 1446, "retrieve_doc": true, "source": "transformers"}, "hash": "4c4d6c9980cba714dac6e501bec0e2b2a66ba7b9107ec4d47a96e2ec585f0ca7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3322, "end_char_idx": 5566, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage example

CANINE works on raw characters, so it can be used **without a tokenizer**:

```python
>>> from transformers import CanineModel
>>> import torch

>>> model = CanineModel.from_pretrained("google/canine-c")  # model pre-trained with autoregressive character loss

>>> text = "hello world"
>>> # use Python's built-in ord() function to turn each character into its unicode code point id
>>> input_ids = torch.tensor([[ord(char) for char in text]])

>>> outputs = model(input_ids)  # forward pass
>>> pooled_output = outputs.pooler_output
>>> sequence_output = outputs.last_hidden_state
```

For batched inference and training, it is however recommended to make use of the tokenizer (to pad/truncate all
sequences to the same length):

```python
>>> from transformers import CanineTokenizer, CanineModel

>>> model = CanineModel.from_pretrained("google/canine-c")
>>> tokenizer = CanineTokenizer.from_pretrained("google/canine-c")

>>> inputs = ["Life is like a box of chocolates.", "You never know what you gonna get."]
>>> encoding = tokenizer(inputs, padding="longest", truncation=True, return_tensors="pt")

>>> outputs = model(**encoding)  # forward pass
>>> pooled_output = outputs.pooler_output
>>> sequence_output = outputs.last_hidden_state
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Multiple choice task guide](../tasks/multiple_choice)

## CanineConfig

[[autodoc]] CanineConfig

## CanineTokenizer

[[autodoc]] CanineTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences

## CANINE specific outputs

[[autodoc]] models.canine.modeling_canine.CanineModelOutputWithPooling

## CanineModel

[[autodoc]] CanineModel
    - forward

## CanineForSequenceClassification

[[autodoc]] CanineForSequenceClassification
    - forward

## CanineForMultipleChoice

[[autodoc]] CanineForMultipleChoice
    - forward

## CanineForTokenClassification

[[autodoc]] CanineForTokenClassification
    - forward

## CanineForQuestionAnswering

[[autodoc]] CanineForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "30d65d37-3d7a-418f-8b44-846d41291653", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bde4f876-d5b9-4f3a-b071-c9860dfb96ba", "node_type": "1", "metadata": {}, "hash": "93be05c5d0cf367ff6ed445162b83a659557faa32e35e1cf703f9b6d6bf38f38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1708, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VipLlava

## Overview

The VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.

VipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a "red bounding box" or "pointed arrow" during training.

The abstract from the paper is the following:

*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*

The original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)


key: _node_content
string_value:
{"id_": "bde4f876-d5b9-4f3a-b071-c9860dfb96ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30d65d37-3d7a-418f-8b44-846d41291653", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "25d652330a465a6dfb45458464a53c5e431e16d85d8a076d195846de4f02915f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03d60dea-5d0d-4b62-80e3-76c31f782ebf", "node_type": "1", "metadata": {}, "hash": "98ee1174bd6aac28765f57e81130931612ca885aaa5c686caabeb6a339670a2f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4336, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VipLlava

## Overview

The VipLlava model was proposed in [Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://arxiv.org/abs/2312.00784) by Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee.

VipLlava enhances the training protocol of Llava by marking images and interact with the model using natural cues like a "red bounding box" or "pointed arrow" during training.

The abstract from the paper is the following:

*While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available.*

The original code can be found [here](https://github.com/mu-cai/ViP-LLaVA).

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada)


## Usage tips:

- The architecture is similar than llava architecture except that the multi-modal projector takes a set of concatenated vision hidden states and has an additional layernorm layer on that module.

- We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating.

- Note the model has not been explicitly trained to process multiple images in the same prompt, although this is technically possible, you may experience inaccurate results.

- For better results, we recommend users to use the processor's `apply_chat_template()` method to format your prompt correctly. For that you need to construct a conversation history, passing in a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys "role" and "content". The "content" should be a list of dictionaries, for "text" and "image" modalities, as follows:

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What’s shown in this image?"},
         ,
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image in more details."},
        ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
>>> "###Human: <image>\nWhat’s shown in this image?###Assistant: This image shows a red stop sign.###Human: Describe the image in more details.###Assistant:"
```

- If you want to construct a chat prompt yourself, below is a list of prompt formats accepted by VipLLaVa checkpoints:
```bash
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\n<prompt>###Assistant:
```

For multiple turns conversation:
```bash
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.###Human: <image>\n<prompt1>###Assistant: <answer1>###Human: <prompt2>###Assistant:
```


key: _node_content
string_value:
{"id_": "03d60dea-5d0d-4b62-80e3-76c31f782ebf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7b5ffa8-6dd8-46a3-b7dd-851625361d77", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "07f9f1dfc1472c71e6a676c5000f98818aa134a11fb06b9a1978798cb75221d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bde4f876-d5b9-4f3a-b071-c9860dfb96ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vipllava", "title": "VipLlava", "tokens": 1159, "retrieve_doc": true, "source": "transformers"}, "hash": "ed15255c288e104d76811a249adbd344afd2c5b4157010b6205ea37e86b444f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4339, "end_char_idx": 4481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## VipLlavaConfig

[[autodoc]] VipLlavaConfig

## VipLlavaForConditionalGeneration

[[autodoc]] VipLlavaForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "a2712ccc-1b6b-4438-8e06-f0b833d738f8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a0c33592-59ae-43b2-bf45-a34a17e0c0b5", "node_type": "1", "metadata": {}, "hash": "4999b2128a4caac8a2de2ceb2add6f3c9773df08a0b09e6befa9d18d535a379a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# NLLB

## Updated tokenizer behavior 

**DISCLAIMER:** The default behaviour for the tokenizer was fixed and thus changed in April 2023.
The previous version adds `[self.eos_token_id, self.cur_lang_code]` at the end of the token sequence for both target and source tokenization. This is wrong as the NLLB paper mentions (page 48, 6.1.1. Model Architecture) :

*Note that we prefix the source sequence with the source language, as opposed to the target
language as previously done in several works (Arivazhagan et al., 2019; Johnson et al.,
2017). This is primarily because we prioritize optimizing zero-shot performance of our
model on any pair of 200 languages at a minor cost to supervised performance.*

Previous behaviour:

```python
>>> from transformers import NllbTokenizer

>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer("How was your day?").input_ids
[13374, 1398, 4260, 4039, 248130, 2, 256047]

>>> # 2: '</s>'
>>> # 256047 : 'eng_Latn'
```
New behaviour

```python
>>> from transformers import NllbTokenizer

>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer("How was your day?").input_ids
[256047, 13374, 1398, 4260, 4039, 248130, 2]
 ```

Enabling the old behaviour can be done as follows:
```python
>>> from transformers import NllbTokenizer

>>> tokenizer = NllbTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", legacy_behaviour=True)
```

For more details, feel free to check the linked [PR](https://github.com/huggingface/transformers/pull/22313) and [Issue](https://github.com/huggingface/transformers/issues/19943).

## Overview

The NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-jussà, James Cross, Onur Çelebi,
Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,
Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,
Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Safiyyah Saleem, Holger Schwenk, and Jeff Wang.

The abstract of the paper is the following:

*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the
200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind?


key: _node_content
string_value:
{"id_": "a0c33592-59ae-43b2-bf45-a34a17e0c0b5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2712ccc-1b6b-4438-8e06-f0b833d738f8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "069e487edb67c2ba49361b3cc923ef8ddf78f05883e7a66e0a11ad69fd3c1d15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "606bf67e-f9ae-4d71-ab1b-4835cdc400bc", "node_type": "1", "metadata": {}, "hash": "c3bc72fbff32006d6c0e81560f7760df9b1d97674b43d8eef427a52dc297efce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1645, "end_char_idx": 4778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Overview

The NLLB model was presented in [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672) by Marta R. Costa-jussà, James Cross, Onur Çelebi,
Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula,
Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews,
Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Safiyyah Saleem, Holger Schwenk, and Jeff Wang.

The abstract of the paper is the following:

*Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the
200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by
first contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed
at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of
Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training
improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using
a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.
Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*

This implementation contains the dense models available on release.

**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**

This model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).

## Generating with NLLB

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.


key: _node_content
string_value:
{"id_": "606bf67e-f9ae-4d71-ab1b-4835cdc400bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a0c33592-59ae-43b2-bf45-a34a17e0c0b5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "599e06f839ed76580ce020084de6d114d2ad4e9de092ce5c268f22d467cbb9a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "78b3d3f4-141c-4c08-99b8-60682823d9e3", "node_type": "1", "metadata": {}, "hash": "33de18180082bae679fafe25b290fa989ff8c5b661bb0089c2b9fd392e34eb3d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3089, "end_char_idx": 5748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then, we created datasets and models aimed
at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of
Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training
improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using
a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety.
Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system.*

This implementation contains the dense models available on release.

**The sparse model NLLB-MoE (Mixture of Expert) is now available! More details [here](nllb-moe)**

This model was contributed by [Lysandre](https://huggingface.co/lysandre). The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).

## Generating with NLLB

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "UN Chief says there is no military solution in Syria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("fra_Latn"), max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
Le chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie
```

### Generating from any other language than English

English (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.


key: _node_content
string_value:
{"id_": "78b3d3f4-141c-4c08-99b8-60682823d9e3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "606bf67e-f9ae-4d71-ab1b-4835cdc400bc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "28878c13e282282d82526530c84df08f43e58f578c6b89d9fce2103f238ba53a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1549202b-6530-4475-9211-b85bae98cd69", "node_type": "1", "metadata": {}, "hash": "2e396380330057478b5db6f533a4ef117313ba5bd5065a58dc42286aae40b797", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4229, "end_char_idx": 7169, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The authors' code can be found [here](https://github.com/facebookresearch/fairseq/tree/nllb).

## Generating with NLLB

While generating the target text set the `forced_bos_token_id` to the target language id. The following
example shows how to translate English to French using the *facebook/nllb-200-distilled-600M* model.

Note that we're using the BCP-47 code for French `fra_Latn`. See [here](https://github.com/facebookresearch/flores/blob/main/flores200/README.md#languages-in-flores-200)
for the list of all BCP-47 in the Flores 200 dataset.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "UN Chief says there is no military solution in Syria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("fra_Latn"), max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
Le chef de l'ONU dit qu'il n'y a pas de solution militaire en Syrie
```

### Generating from any other language than English

English (`eng_Latn`) is set as the default language from which to translate. In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.

See example below for a translation from romanian to german:

```py
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained(
...     "facebook/nllb-200-distilled-600M", token=True, src_lang="ron_Latn"
... )
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", token=True)

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("deu_Latn"), max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
UN-Chef sagt, es gibt keine militärische Lösung in Syrien
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## NllbTokenizer

[[autodoc]] NllbTokenizer
    - build_inputs_with_special_tokens

## NllbTokenizerFast

[[autodoc]] NllbTokenizerFast

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).


key: _node_content
string_value:
{"id_": "1549202b-6530-4475-9211-b85bae98cd69", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78b3d3f4-141c-4c08-99b8-60682823d9e3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "cc071fa82e71d18ce57049ab00c6ebd9ada0c0a49e423a24e074da5411fa903d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "266cd43f-7f84-48dc-8eb7-ce102d96b07d", "node_type": "1", "metadata": {}, "hash": "70d3c6af1abb7ba92b9151bb8c5b3520351c73b33dbd88c3b8f70fa3864f3a27", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5570, "end_char_idx": 8610, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In order to specify that you'd like to translate from a different language,
you should specify the BCP-47 code in the `src_lang` keyword argument of the tokenizer initialization.

See example below for a translation from romanian to german:

```py
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained(
...     "facebook/nllb-200-distilled-600M", token=True, src_lang="ron_Latn"
... )
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", token=True)

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("deu_Latn"), max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
UN-Chef sagt, es gibt keine militärische Lösung in Syrien
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## NllbTokenizer

[[autodoc]] NllbTokenizer
    - build_inputs_with_special_tokens

## NllbTokenizerFast

[[autodoc]] NllbTokenizerFast

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.

```python
>>> import torch
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda").eval()
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt").to("cuda")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("deu_Latn"), max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"UN-Chef sagt, es gibt keine militärische Lösung in Syrien"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.


key: _node_content
string_value:
{"id_": "266cd43f-7f84-48dc-8eb7-ce102d96b07d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a548662-ff11-4b4f-8b7e-0d36729dea0a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "2bfe8515d58245cd9b09950479b511e8896843df8a89f9ff8d60f2e237078321", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1549202b-6530-4475-9211-b85bae98cd69", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nllb", "title": "NLLB", "tokens": 2326, "retrieve_doc": true, "source": "transformers"}, "hash": "453531d98e92fcbdf376a917a2584b1439791ab420c159674a55e12568859f39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7013, "end_char_idx": 8758, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.

```python
>>> import torch
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda").eval()
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt").to("cuda")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("deu_Latn"), max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"UN-Chef sagt, es gibt keine militärische Lösung in Syrien"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp">
</div>


key: _node_content
string_value:
{"id_": "1bc266f3-0f89-47a1-918c-aec9f5373169", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "113a5b02-de4c-4cf9-bff9-4bb6899ee481", "node_type": "1", "metadata": {}, "hash": "f39a0e5529512db7b56c4c5e69f1d32a58ab45bc145f1178d7a17b01bf978729", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3321, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DeiT

## Overview

The DeiT model was proposed in [Training data-efficient image transformers & distillation through attention](https://arxiv.org/abs/2012.12877) by Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, Hervé Jégou. The [Vision Transformer (ViT)](vit) introduced in [Dosovitskiy et al., 2020](https://arxiv.org/abs/2010.11929) has shown that one can match or even outperform existing convolutional neural
networks using a Transformer encoder (BERT-like). However, the ViT models introduced in that paper required training on
expensive infrastructure for multiple weeks, using external data. DeiT (data-efficient image transformers) are more
efficiently trained transformers for image classification, requiring far less data and far less computing resources
compared to the original ViT models.

The abstract from the paper is the following:

*Recently, neural networks purely based on attention were shown to address image understanding tasks such as image
classification. However, these visual transformers are pre-trained with hundreds of millions of images using an
expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free
transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision
transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external
data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation
token ensuring that the student learns from the teacher through attention. We show the interest of this token-based
distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets
for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and
models.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).

## Usage tips

- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the
  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with
  the class ([CLS]) and patch tokens through the self-attention layers.
- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top
  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a
  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction
  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the
  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the
  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction
  between both heads as final prediction. (2) is also called "fine-tuning with distillation", because one relies on a
  teacher that has already been fine-tuned on the downstream dataset.


key: _node_content
string_value:
{"id_": "113a5b02-de4c-4cf9-bff9-4bb6899ee481", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1bc266f3-0f89-47a1-918c-aec9f5373169", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "441e3e7de384fe5762793e113bab4cd0119e40f4193b67866e01a082afa87d59", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4ab58df-a62a-4448-8c35-a89740fd27d2", "node_type": "1", "metadata": {}, "hash": "8900adc0324e373391c6583eb4712e15c41d06e2414db67c22b253bde705edbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1662, "end_char_idx": 4842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We show the interest of this token-based
distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets
for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and
models.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts).

## Usage tips

- Compared to ViT, DeiT models use a so-called distillation token to effectively learn from a teacher (which, in the
  DeiT paper, is a ResNet like-model). The distillation token is learned through backpropagation, by interacting with
  the class ([CLS]) and patch tokens through the self-attention layers.
- There are 2 ways to fine-tune distilled models, either (1) in a classic way, by only placing a prediction head on top
  of the final hidden state of the class token and not using the distillation signal, or (2) by placing both a
  prediction head on top of the class token and on top of the distillation token. In that case, the [CLS] prediction
  head is trained using regular cross-entropy between the prediction of the head and the ground-truth label, while the
  distillation prediction head is trained using hard distillation (cross-entropy between the prediction of the
  distillation head and the label predicted by the teacher). At inference time, one takes the average prediction
  between both heads as final prediction. (2) is also called "fine-tuning with distillation", because one relies on a
  teacher that has already been fine-tuned on the downstream dataset. In terms of models, (1) corresponds to
  [`DeiTForImageClassification`] and (2) corresponds to
  [`DeiTForImageClassificationWithTeacher`].
- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is
  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.
- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in
  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for
  pre-training.
- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into
  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data
  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset
  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):
  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and
  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to
  prepare images for the model.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use.


key: _node_content
string_value:
{"id_": "e4ab58df-a62a-4448-8c35-a89740fd27d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "113a5b02-de4c-4cf9-bff9-4bb6899ee481", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "a61caf16365eadfd663d915a8e69dfb8fccaf9e99ea1b4c687b852fe55c757e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee39c02b-8fd3-4e87-b4f5-3267899750d3", "node_type": "1", "metadata": {}, "hash": "029ca90e3a085074fd8493a422be77182cdb82522184983bf5b3107400b982dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3322, "end_char_idx": 5839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In terms of models, (1) corresponds to
  [`DeiTForImageClassification`] and (2) corresponds to
  [`DeiTForImageClassificationWithTeacher`].
- Note that the authors also did try soft distillation for (2) (in which case the distillation prediction head is
  trained using KL divergence to match the softmax output of the teacher), but hard distillation gave the best results.
- All released checkpoints were pre-trained and fine-tuned on ImageNet-1k only. No external data was used. This is in
  contrast with the original ViT model, which used external data like the JFT-300M dataset/Imagenet-21k for
  pre-training.
- The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into
  [`ViTModel`] or [`ViTForImageClassification`]. Techniques like data
  augmentation, optimization, and regularization were used in order to simulate training on a much larger dataset
  (while only using ImageNet-1k for pre-training). There are 4 variants available (in 3 different sizes):
  *facebook/deit-tiny-patch16-224*, *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and
  *facebook/deit-base-patch16-384*. Note that one should use [`DeiTImageProcessor`] in order to
  prepare images for the model.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import DeiTForImageClassification
model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.


key: _node_content
string_value:
{"id_": "ee39c02b-8fd3-4e87-b4f5-3267899750d3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4ab58df-a62a-4448-8c35-a89740fd27d2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "667d94f09a853d5e030af32ed35b1fed8b792b55bbc22602707dafaa533d9d5b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "911713fe-975f-4e3f-a087-18bd0f55b821", "node_type": "1", "metadata": {}, "hash": "48d04040b93eee91b93afc81ec5bba08c29cfde39234bf8ac7a8f2bfa36633dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4478, "end_char_idx": 7589, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that one should use [`DeiTImageProcessor`] in order to
  prepare images for the model.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import DeiTForImageClassification
model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         8 |                                         6 |                      1.33 |
|            2 |                                         9 |                                         6 |                      1.5  |
|            4 |                                         9 |                                         6 |                      1.5  |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DeiT.

<PipelineTag pipeline="image-classification"/>

- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

Besides that:

- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "911713fe-975f-4e3f-a087-18bd0f55b821", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "540f096f-67a0-4958-bbbc-3db91b5ac24d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5619f19de12a1b3e8f588745eb38e816eb35c4884c8b5951de149ca446e346", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee39c02b-8fd3-4e87-b4f5-3267899750d3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deit", "title": "DeiT", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cd83813f908408df4533323524bdc6c90912a0f6c096b10602640297998e3b62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5658, "end_char_idx": 8498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `facebook/deit-base-distilled-patch16-224` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         8 |                                         6 |                      1.33 |
|            2 |                                         9 |                                         6 |                      1.5  |
|            4 |                                         9 |                                         6 |                      1.5  |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DeiT.

<PipelineTag pipeline="image-classification"/>

- [`DeiTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

Besides that:

- [`DeiTForMaskedImageModeling`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DeiTConfig

[[autodoc]] DeiTConfig

## DeiTFeatureExtractor

[[autodoc]] DeiTFeatureExtractor
    - __call__

## DeiTImageProcessor

[[autodoc]] DeiTImageProcessor
    - preprocess

<frameworkcontent>
<pt>

## DeiTModel

[[autodoc]] DeiTModel
    - forward

## DeiTForMaskedImageModeling

[[autodoc]] DeiTForMaskedImageModeling
    - forward

## DeiTForImageClassification

[[autodoc]] DeiTForImageClassification
    - forward

## DeiTForImageClassificationWithTeacher

[[autodoc]] DeiTForImageClassificationWithTeacher
    - forward

</pt>
<tf>

## TFDeiTModel

[[autodoc]] TFDeiTModel
    - call

## TFDeiTForMaskedImageModeling

[[autodoc]] TFDeiTForMaskedImageModeling
    - call

## TFDeiTForImageClassification

[[autodoc]] TFDeiTForImageClassification
    - call

## TFDeiTForImageClassificationWithTeacher

[[autodoc]] TFDeiTForImageClassificationWithTeacher
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "f4326f60-a7fa-4a35-a074-025a472c2fae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pegasus_x", "title": "PEGASUS-X", "tokens": 630, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "38ec819c-1a97-4e8c-ac50-135704a23e0d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pegasus_x", "title": "PEGASUS-X", "tokens": 630, "retrieve_doc": true, "source": "transformers"}, "hash": "2ecb7f0c047d782b936dfcdeac84b858c4d7fa5cd16967f6d4028bd9be2f1789", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2134, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PEGASUS-X

## Overview

The PEGASUS-X model was proposed in [Investigating Efficiently Extending Transformers for Long Input Summarization](https://arxiv.org/abs/2208.04347)  by Jason Phang, Yao Zhao and Peter J. Liu.

PEGASUS-X (PEGASUS eXtended) extends the PEGASUS models for long input summarization through additional long input pretraining and using staggered block-local attention with global tokens in the encoder.

The abstract from the paper is the following:

*While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most pretrained models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms can most efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. PEGASUS-X achieves strong performance on long input summarization tasks comparable with much larger models while adding few additional parameters and not requiring model parallelism to train.*

This model was contributed by [zphang](https://huggingface.co/zphang). The original code can be found [here](https://github.com/google-research/pegasus).

## Documentation resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

<Tip>

PEGASUS-X uses the same tokenizer as [PEGASUS](pegasus).

</Tip>

## PegasusXConfig

[[autodoc]] PegasusXConfig

## PegasusXModel

[[autodoc]] PegasusXModel
    - forward

## PegasusXForConditionalGeneration

[[autodoc]] PegasusXForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "9b25f1ff-b231-400b-a84b-1d3a52eafdc1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2650753e-bea2-4c35-baff-7795ba332a33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db87bad8-5088-4b18-bf91-d5b484e9c133", "node_type": "1", "metadata": {}, "hash": "65d57d19aec87f1f8f24b08ef79d17237774788559ef0021cc1da7a25f288e53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3857, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Wav2Vec2-BERT

## Overview

The Wav2Vec2-BERT model was proposed in [Seamless: Multilingual Expressive and Streaming Speech Translation](https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/) by the Seamless Communication team from Meta AI.

This model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification.

The official results of the model can be found in Section 3.2.1 of the paper.

The abstract from the paper is the following:

*Recent advancements in automatic speech translation have dramatically expanded language coverage, improved multimodal capabilities, and enabled a wide range of tasks and functionalities. That said, large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model—SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. The expanded version of SeamlessAlign adds 114,800 hours of automatically aligned data for a total of 76 languages. SeamlessM4T v2 provides the foundation on which our two newest models, SeamlessExpressive and SeamlessStreaming, are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one’s voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work—including models, code, and a watermark detector—are publicly released and accessible at the link below.*

This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## Usage tips

- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.


key: _node_content
string_value:
{"id_": "db87bad8-5088-4b18-bf91-d5b484e9c133", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2650753e-bea2-4c35-baff-7795ba332a33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b25f1ff-b231-400b-a84b-1d3a52eafdc1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "2b0d106a5362f899f9b16eebdff7cc010e29d965dcb6f6e925ef321302577821", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "968606fc-31d1-4116-9ddd-ac21c95be431", "node_type": "1", "metadata": {}, "hash": "2e633c2b946c447014c7daa89f3b7aabd29d91ce849309fba975fbcf4b232c52", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1906, "end_char_idx": 5121, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention (EMMA) mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To understand the performance of these models, we combined novel and modified versions of existing automatic metrics to evaluate prosody, latency, and robustness. For human evaluations, we adapted existing protocols tailored for measuring the most relevant attributes in the preservation of meaning, naturalness, and expressivity. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. In sum, Seamless gives us a pivotal look at the technical foundation needed to turn the Universal Speech Translator from a science fiction concept into a real-world technology. Finally, contributions in this work—including models, code, and a watermark detector—are publicly released and accessible at the link below.*

This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## Usage tips

- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.
- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or
  rotary position embeddings by setting the correct `config.position_embeddings_type`.
- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.

## Resources

<PipelineTag pipeline="automatic-speech-recognition"/>

- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).
- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).

<PipelineTag pipeline="audio-classification"/>

- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).
- See also: [Audio classification task guide](../tasks/audio_classification)


key: _node_content
string_value:
{"id_": "968606fc-31d1-4116-9ddd-ac21c95be431", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2650753e-bea2-4c35-baff-7795ba332a33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "76199f83168ac684935b57258de857f533f2c2aac536894f6087d91de6052432", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db87bad8-5088-4b18-bf91-d5b484e9c133", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2-bert", "title": "Wav2Vec2-BERT", "tokens": 1431, "retrieve_doc": true, "source": "transformers"}, "hash": "7dff3bf64c6c2812e1f5de426679ae9d826cc48c860730fff49a94583e87dab1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3452, "end_char_idx": 5770, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## Usage tips

- Wav2Vec2-BERT follows the same architecture as Wav2Vec2-Conformer, but employs a causal depthwise convolutional layer and uses as input a mel-spectrogram representation of the audio instead of the raw waveform.
- Wav2Vec2-BERT can use either no relative position embeddings, Shaw-like position embeddings, Transformer-XL-like position embeddings, or
  rotary position embeddings by setting the correct `config.position_embeddings_type`.
- Wav2Vec2-BERT also introduces a Conformer-based adapter network instead of a simple convolutional network.

## Resources

<PipelineTag pipeline="automatic-speech-recognition"/>

- [`Wav2Vec2BertForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition).
- You can also adapt these notebooks on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).

<PipelineTag pipeline="audio-classification"/>

- [`Wav2Vec2BertForSequenceClassification`] can be used by adapting this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification).
- See also: [Audio classification task guide](../tasks/audio_classification)


## Wav2Vec2BertConfig

[[autodoc]] Wav2Vec2BertConfig

## Wav2Vec2BertProcessor

[[autodoc]] Wav2Vec2BertProcessor
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## Wav2Vec2BertModel

[[autodoc]] Wav2Vec2BertModel
    - forward

## Wav2Vec2BertForCTC

[[autodoc]] Wav2Vec2BertForCTC
    - forward

## Wav2Vec2BertForSequenceClassification

[[autodoc]] Wav2Vec2BertForSequenceClassification
    - forward

## Wav2Vec2BertForAudioFrameClassification

[[autodoc]] Wav2Vec2BertForAudioFrameClassification
    - forward

## Wav2Vec2BertForXVector

[[autodoc]] Wav2Vec2BertForXVector
    - forward


key: _node_content
string_value:
{"id_": "5cb4c880-3957-49da-a3b0-c4fcfd2c12f0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "hash": "17479d3a217cd878a98f8600f35ecc4d0c3d51fd8eb0402d29764483afabb83b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "064fd466-830b-433b-99a1-014ae163ec38", "node_type": "1", "metadata": {}, "hash": "ffab6bcf96998c282e8a43a358cf36503f958dc381f5c1735009fae9a74a59e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1700, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoFormer

## Overview

The RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.

The abstract from the paper is the following:

*Position encoding in transformer architecture provides supervision for dependency modeling between elements at
different positions in the sequence. We investigate various methods to encode positional information in
transformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The
proposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative
position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of
being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and
capability of equipping the linear self-attention with relative position encoding. As a result, the enhanced
transformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We
release the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing
experiment for English benchmark will soon be updated.*

This model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).

## Usage tips
RoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown 
improved performance on classification tasks with long texts.


key: _node_content
string_value:
{"id_": "064fd466-830b-433b-99a1-014ae163ec38", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97aba92d-60a2-42d5-bea8-f97e614103fa", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "hash": "17479d3a217cd878a98f8600f35ecc4d0c3d51fd8eb0402d29764483afabb83b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5cb4c880-3957-49da-a3b0-c4fcfd2c12f0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/roformer", "title": "RoFormer", "tokens": 1190, "retrieve_doc": true, "source": "transformers"}, "hash": "b7834f109bb1166e071a7b3456f8211ab4d7fa7a96438056e035befca1299ddb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4211, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RoFormer

## Overview

The RoFormer model was proposed in [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864v1.pdf) by Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.

The abstract from the paper is the following:

*Position encoding in transformer architecture provides supervision for dependency modeling between elements at
different positions in the sequence. We investigate various methods to encode positional information in
transformer-based language models and propose a novel implementation named Rotary Position Embedding(RoPE). The
proposed RoPE encodes absolute positional information with rotation matrix and naturally incorporates explicit relative
position dependency in self-attention formulation. Notably, RoPE comes with valuable properties such as flexibility of
being expand to any sequence lengths, decaying inter-token dependency with increasing relative distances, and
capability of equipping the linear self-attention with relative position encoding. As a result, the enhanced
transformer with rotary position embedding, or RoFormer, achieves superior performance in tasks with long texts. We
release the theoretical analysis along with some preliminary experiment results on Chinese data. The undergoing
experiment for English benchmark will soon be updated.*

This model was contributed by [junnyu](https://huggingface.co/junnyu). The original code can be found [here](https://github.com/ZhuiyiTechnology/roformer).

## Usage tips
RoFormer is a BERT-like autoencoding model with rotary position embeddings. Rotary position embeddings have shown 
improved performance on classification tasks with long texts.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## RoFormerConfig

[[autodoc]] RoFormerConfig

## RoFormerTokenizer

[[autodoc]] RoFormerTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## RoFormerTokenizerFast

[[autodoc]] RoFormerTokenizerFast
    - build_inputs_with_special_tokens

<frameworkcontent>
<pt>

## RoFormerModel

[[autodoc]] RoFormerModel
    - forward

## RoFormerForCausalLM

[[autodoc]] RoFormerForCausalLM
    - forward

## RoFormerForMaskedLM

[[autodoc]] RoFormerForMaskedLM
    - forward

## RoFormerForSequenceClassification

[[autodoc]] RoFormerForSequenceClassification
    - forward

## RoFormerForMultipleChoice

[[autodoc]] RoFormerForMultipleChoice
    - forward

## RoFormerForTokenClassification

[[autodoc]] RoFormerForTokenClassification
    - forward

## RoFormerForQuestionAnswering

[[autodoc]] RoFormerForQuestionAnswering
    - forward

</pt>
<tf>

## TFRoFormerModel

[[autodoc]] TFRoFormerModel
    - call

## TFRoFormerForMaskedLM

[[autodoc]] TFRoFormerForMaskedLM
    - call

## TFRoFormerForCausalLM

[[autodoc]] TFRoFormerForCausalLM
    - call

## TFRoFormerForSequenceClassification

[[autodoc]] TFRoFormerForSequenceClassification
    - call

## TFRoFormerForMultipleChoice

[[autodoc]] TFRoFormerForMultipleChoice
    - call

## TFRoFormerForTokenClassification

[[autodoc]] TFRoFormerForTokenClassification
    - call

## TFRoFormerForQuestionAnswering

[[autodoc]] TFRoFormerForQuestionAnswering
    - call

</tf>
<jax>

## FlaxRoFormerModel

[[autodoc]] FlaxRoFormerModel
    - __call__

## FlaxRoFormerForMaskedLM

[[autodoc]] FlaxRoFormerForMaskedLM
    - __call__

## FlaxRoFormerForSequenceClassification

[[autodoc]] FlaxRoFormerForSequenceClassification
    - __call__

## FlaxRoFormerForMultipleChoice

[[autodoc]] FlaxRoFormerForMultipleChoice
    - __call__

## FlaxRoFormerForTokenClassification

[[autodoc]] FlaxRoFormerForTokenClassification
    - __call__

## FlaxRoFormerForQuestionAnswering

[[autodoc]] FlaxRoFormerForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "78724783-2b89-4d67-892c-40b70bc7f3ce", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b9480d4d-4b21-4c51-b1a2-50394c6c035f", "node_type": "1", "metadata": {}, "hash": "81e302000b73b7f8ce2ec8be03622b6551b8faebac0a5553ec089d84c0e61ac6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3421, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OWL-ViT

## Overview

The OWL-ViT (short for Vision Transformer for Open-World Localization) was proposed in [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230) by Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. OWL-ViT is an open-vocabulary object detection network trained on a variety of (image, text) pairs. It can be used to query an image with one or multiple text queries to search for and detect target objects described in text.

The abstract from the paper is the following:

*Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlvit_architecture.jpg"
alt="drawing" width="600"/>

<small> OWL-ViT architecture. Taken from the <a href="https://arxiv.org/abs/2205.06230">original paper</a>. </small>

This model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage tips

OWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`].


key: _node_content
string_value:
{"id_": "b9480d4d-4b21-4c51-b1a2-50394c6c035f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78724783-2b89-4d67-892c-40b70bc7f3ce", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "8c4bcb4ebe89b6a230fbd6f613aaea9a7fb9e195fca451dfc4ded5821d59ef12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0f216e4-e89d-484f-8e57-36d5612115f3", "node_type": "1", "metadata": {}, "hash": "0b1eb6de101be3d103adb3b894974e3eec3d64ad5f84861710a948747eb302af", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1881, "end_char_idx": 5186, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2205.06230">original paper</a>. </small>

This model was contributed by [adirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage tips

OWL-ViT is a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`OwlViTImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`OwlViTProcessor`] wraps [`OwlViTImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`OwlViTProcessor`] and [`OwlViTForObjectDetection`].

```python
>>> import requests
>>> from PIL import Image
>>> import torch

>>> from transformers import OwlViTProcessor, OwlViTForObjectDetection

>>> processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
>>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> texts = [["a photo of a cat", "a photo of a dog"]]
>>> inputs = processor(text=texts, images=image, return_tensors="pt")
>>> outputs = model(**inputs)

>>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]
>>> target_sizes = torch.Tensor([image.size[::-1]])
>>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)
>>> results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)
>>> i = 0  # Retrieve predictions for the first image for the corresponding text queries
>>> text = texts[i]
>>> boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
>>> for box, score, label in zip(boxes, scores, labels):
...     box = [round(i, 2) for i in box.tolist()]
...     print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}")
Detected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29]
Detected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17]
```

## Resources

A demo notebook on using OWL-ViT for zero- and one-shot (image-guided) object detection can be found [here](https://github.com/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb).


key: _node_content
string_value:
{"id_": "c0f216e4-e89d-484f-8e57-36d5612115f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e5e5ee0-fd35-4b10-902b-a535875dfe30", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "b6e34bf2940493b31335054f63c3e644a3eeb7c581e0f8e60cf899b66ac57329", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b9480d4d-4b21-4c51-b1a2-50394c6c035f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlvit", "title": "OWL-ViT", "tokens": 1628, "retrieve_doc": true, "source": "transformers"}, "hash": "069228ac6e7f32ee56fc606c95a0475003183ae4624651071b9ce1e1f790ee8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5188, "end_char_idx": 6052, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## OwlViTConfig

[[autodoc]] OwlViTConfig
    - from_text_vision_configs

## OwlViTTextConfig

[[autodoc]] OwlViTTextConfig

## OwlViTVisionConfig

[[autodoc]] OwlViTVisionConfig

## OwlViTImageProcessor

[[autodoc]] OwlViTImageProcessor
    - preprocess
    - post_process_object_detection
    - post_process_image_guided_detection

## OwlViTFeatureExtractor

[[autodoc]] OwlViTFeatureExtractor
    - __call__
    - post_process
    - post_process_image_guided_detection

## OwlViTProcessor

[[autodoc]] OwlViTProcessor

## OwlViTModel

[[autodoc]] OwlViTModel
    - forward
    - get_text_features
    - get_image_features

## OwlViTTextModel

[[autodoc]] OwlViTTextModel
    - forward

## OwlViTVisionModel

[[autodoc]] OwlViTVisionModel
    - forward

## OwlViTForObjectDetection

[[autodoc]] OwlViTForObjectDetection
    - forward
    - image_guided_detection


key: _node_content
string_value:
{"id_": "0953080d-064d-4161-afa8-465c0ce4be27", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4fc3f7f9-8551-4bb9-b65e-12a623e04555", "node_type": "1", "metadata": {}, "hash": "f8d1190655f679eb1475dd006586be6f3ff80afc08dcd97ee8696cf225586bda", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RWKV

## Overview

The RWKV model was proposed in [this repo](https://github.com/BlinkDL/RWKV-LM)

It suggests a tweak in the traditional Transformer attention to make it linear. This way, the model can be used as recurrent network: passing inputs for timestamp 0 and timestamp 1 together is the same as passing inputs at timestamp 0, then inputs at timestamp 1 along with the state of timestamp 0 (see example below).

This can be more efficient than a regular Transformer and can deal with sentence of any length (even if the model uses a fixed context length for training).

This model was contributed by [sgugger](https://huggingface.co/sgugger).
The original code can be found [here](https://github.com/BlinkDL/RWKV-LM).

## Usage example

```py
import torch
from transformers import AutoTokenizer, RwkvConfig, RwkvModel

model = RwkvModel.from_pretrained("sgugger/rwkv-430M-pile")
tokenizer = AutoTokenizer.from_pretrained("sgugger/rwkv-430M-pile")

inputs = tokenizer("This is an example.", return_tensors="pt")
# Feed everything to the model
outputs = model(inputs["input_ids"])
output_whole = outputs.last_hidden_state

outputs = model(inputs["input_ids"][:, :2])
output_one = outputs.last_hidden_state

# Using the state computed on the first inputs, we will get the same output
outputs = model(inputs["input_ids"][:, 2:], state=outputs.state)
output_two = outputs.last_hidden_state

torch.allclose(torch.cat([output_one, output_two], dim=1), output_whole, atol=1e-5)
```

If you want to make sure the model stops generating when `'\n\n'` is detected, we recommend using the following stopping criteria:

```python 
from transformers import StoppingCriteria

class RwkvStoppingCriteria(StoppingCriteria):
    def __init__(self, eos_sequence = [187,187], eos_token_id = 537):
        self.eos_sequence = eos_sequence
        self.eos_token_id = eos_token_id

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        last_2_ids = input_ids[:,-2:].tolist()
        return self.eos_sequence in last_2_ids


output = model.generate(inputs["input_ids"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])
```

## RwkvConfig

[[autodoc]] RwkvConfig

## RwkvModel

[[autodoc]] RwkvModel
    - forward

## RwkvLMHeadModel

[[autodoc]] RwkvForCausalLM
    - forward

## Rwkv attention and the recurrent formulas

In a traditional auto-regressive Transformer, attention is written as

$$O = \hbox{softmax}(QK^{T} / \sqrt{d}) V$$

with \\(Q\\), \\(K\\) and \\(V\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\(QK^{T}\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\(V\\) to get the output \\(O\\) of the same shape as the others.


key: _node_content
string_value:
{"id_": "4fc3f7f9-8551-4bb9-b65e-12a623e04555", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0953080d-064d-4161-afa8-465c0ce4be27", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "735511b3960569bb826cedaa9a89fa62fbcf1af733529378c5ea4b4abc2ef438", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05dc3fe4-8ead-4c92-aa06-283f585babec", "node_type": "1", "metadata": {}, "hash": "45d12045c247d8d9d94d6b0070a8be57312cf7cc67a0497b44465a1bb9e6efb2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2064, "end_char_idx": 3986, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
output = model.generate(inputs["input_ids"], max_new_tokens=64, stopping_criteria = [RwkvStoppingCriteria()])
```

## RwkvConfig

[[autodoc]] RwkvConfig

## RwkvModel

[[autodoc]] RwkvModel
    - forward

## RwkvLMHeadModel

[[autodoc]] RwkvForCausalLM
    - forward

## Rwkv attention and the recurrent formulas

In a traditional auto-regressive Transformer, attention is written as

$$O = \hbox{softmax}(QK^{T} / \sqrt{d}) V$$

with \\(Q\\), \\(K\\) and \\(V\\) are matrices of shape `seq_len x hidden_size` named query, key and value (they are actually bigger matrices with a batch dimension and an attention head dimension but we're only interested in the last two, which is where the matrix product is taken, so for the sake of simplicity we only consider those two). The product \\(QK^{T}\\) then has shape `seq_len x seq_len` and we can take the matrix product with \\(V\\) to get the output \\(O\\) of the same shape as the others.  

Replacing the softmax by its value gives:

$$O_{i} = \frac{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}} V_{j}}{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}}}$$

Note that the entries in \\(QK^{T}\\) corresponding to \\(j > i\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).

In comparison, the RWKV attention is given by

$$O_{i} = \sigma(R_{i}) \frac{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$

where \\(R\\) is a new matrix called receptance by the author, \\(K\\) and \\(V\\) are still the key and value (\\(\sigma\\) here is the sigmoid function). \\(W\\) is a new vector that represents the position of the token and is given by

$$W_{0} = u \hbox{  and  } W_{k} = (k-1)w \hbox{ for } k \geq 1$$

with \\(u\\) and \\(w\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively.


key: _node_content
string_value:
{"id_": "05dc3fe4-8ead-4c92-aa06-283f585babec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4fc3f7f9-8551-4bb9-b65e-12a623e04555", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "8d1b70b95ef83f40dbcb4b0ce55acc38d535591b803fb44be8fb4a8c0e7b474d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "838d9526-2396-4978-8856-b7d108ba1bed", "node_type": "1", "metadata": {}, "hash": "9a4394f13ca6b85438cddb0baecbdb6c1cf8dcd382f551326a7abc2c410d054a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3007, "end_char_idx": 4787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Replacing the softmax by its value gives:

$$O_{i} = \frac{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}} V_{j}}{\sum_{j=1}^{i} e^{Q_{i} K_{j}^{T} / \sqrt{d}}}$$

Note that the entries in \\(QK^{T}\\) corresponding to \\(j > i\\) are masked (the sum stops at j) because the attention is not allowed to look at future tokens (only past ones).

In comparison, the RWKV attention is given by

$$O_{i} = \sigma(R_{i}) \frac{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}} V_{j}}{\sum_{j=1}^{i} e^{W_{i-j} + K_{j}}}$$

where \\(R\\) is a new matrix called receptance by the author, \\(K\\) and \\(V\\) are still the key and value (\\(\sigma\\) here is the sigmoid function). \\(W\\) is a new vector that represents the position of the token and is given by

$$W_{0} = u \hbox{  and  } W_{k} = (k-1)w \hbox{ for } k \geq 1$$

with \\(u\\) and \\(w\\) learnable parameters called in the code `time_first` and `time_decay` respectively. The numerator and denominator can both be expressed recursively. Naming them \\(N_{i}\\) and \\(D_{i}\\) we have:

$$N_{i} = e^{u + K_{i}} V_{i} + \hat{N}_{i} \hbox{  where  } \hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \cdots + e^{(i-2)w + K_{1}} V_{1}$$

so \\(\hat{N}_{i}\\) (called `numerator_state` in the code) satisfies

$$\hat{N}_{0} = 0 \hbox{  and  } \hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \hat{N}_{j}$$

and

$$D_{i} = e^{u + K_{i}} + \hat{D}_{i} \hbox{  where  } \hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \cdots + e^{(i-2)w + K_{1}}$$

so \\(\hat{D}_{i}\\) (called `denominator_state` in the code) satisfies

$$\hat{D}_{0} = 0 \hbox{  and  } \hat{D}_{j+1} = e^{K_{j}} + e^{w} \hat{D}_{j}$$

The actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers.


key: _node_content
string_value:
{"id_": "838d9526-2396-4978-8856-b7d108ba1bed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05dc3fe4-8ead-4c92-aa06-283f585babec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "b8f242904abfe0f180319d22a9ee88217e017df62cfbf4d87d71b5c8c3f16f9e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "124e3432-6ac4-421f-aabb-4398174006b9", "node_type": "1", "metadata": {}, "hash": "131f8a39cbec6c5f7e458881eab01c3e4c0255e2444cc44b2aecc9c64db9c021", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3922, "end_char_idx": 5245, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The numerator and denominator can both be expressed recursively. Naming them \\(N_{i}\\) and \\(D_{i}\\) we have:

$$N_{i} = e^{u + K_{i}} V_{i} + \hat{N}_{i} \hbox{  where  } \hat{N}_{i} = e^{K_{i-1}} V_{i-1} + e^{w + K_{i-2}} V_{i-2} \cdots + e^{(i-2)w + K_{1}} V_{1}$$

so \\(\hat{N}_{i}\\) (called `numerator_state` in the code) satisfies

$$\hat{N}_{0} = 0 \hbox{  and  } \hat{N}_{j+1} = e^{K_{j}} V_{j} + e^{w} \hat{N}_{j}$$

and

$$D_{i} = e^{u + K_{i}} + \hat{D}_{i} \hbox{  where  } \hat{D}_{i} = e^{K_{i-1}} + e^{w + K_{i-2}} \cdots + e^{(i-2)w + K_{1}}$$

so \\(\hat{D}_{i}\\) (called `denominator_state` in the code) satisfies

$$\hat{D}_{0} = 0 \hbox{  and  } \hat{D}_{j+1} = e^{K_{j}} + e^{w} \hat{D}_{j}$$

The actual recurrent formula used are a tiny bit more complex, as for numerical stability we don't want to compute exponentials of big numbers. Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:

$$\frac{e^{x_{i}}}{\sum_{j=1}^{n} e^{x_{j}}} = \frac{e^{x_{i} - M}}{\sum_{j=1}^{n} e^{x_{j} - M}}$$

with \\(M\\) the maximum of all \\(x_{j}\\). So here on top of saving the numerator state (\\(\hat{N}\\)) and the denominator state (\\(\hat{D}\\)) we also keep track of the maximum of all terms encountered in the exponentials.


key: _node_content
string_value:
{"id_": "124e3432-6ac4-421f-aabb-4398174006b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "14ec845c-6a7d-45c0-88c7-e35436a441c9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "3bacb966fdc0bfa53f5976446af703cbd85b949e2283109b598be93c2d0ae83d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "838d9526-2396-4978-8856-b7d108ba1bed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rwkv", "title": "RWKV", "tokens": 2201, "retrieve_doc": true, "source": "transformers"}, "hash": "26ec72e9d3b6f5221ee6753093cdf86f43ad651ac91c830c60bf3a3e99d08b23", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4788, "end_char_idx": 6055, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Usually the softmax is not computed as is, but the exponential of the maximum term is divided of the numerator and denominator:

$$\frac{e^{x_{i}}}{\sum_{j=1}^{n} e^{x_{j}}} = \frac{e^{x_{i} - M}}{\sum_{j=1}^{n} e^{x_{j} - M}}$$

with \\(M\\) the maximum of all \\(x_{j}\\). So here on top of saving the numerator state (\\(\hat{N}\\)) and the denominator state (\\(\hat{D}\\)) we also keep track of the maximum of all terms encountered in the exponentials. So we actually use

$$\tilde{N}_{i} = e^{-M_{i}} \hat{N}_{i} \hbox{  and  } \tilde{D}_{i} = e^{-M_{i}} \hat{D}_{i}$$

defined by the following recurrent formulas:

$$\tilde{N}_{0} = 0 \hbox{  and  } \tilde{N}_{j+1} = e^{K_{j} - q} V_{j} + e^{w + M_{j} - q} \tilde{N}_{j} \hbox{  where  } q = \max(K_{j}, w + M_{j})$$

and

$$\tilde{D}_{0} = 0 \hbox{  and  } \tilde{D}_{j+1} = e^{K_{j} - q} + e^{w + M_{j} - q} \tilde{D}_{j} \hbox{  where  } q = \max(K_{j}, w + M_{j})$$

and \\(M_{j+1} = q\\). With those, we can then compute

$$N_{i} = e^{u + K_{i} - q} V_{i} + e^{M_{i}} \tilde{N}_{i} \hbox{  where  } q = \max(u + K_{i}, M_{i})$$

and

$$D_{i} = e^{u + K_{i} - q} + e^{M_{i}} \tilde{D}_{i} \hbox{  where  } q = \max(u + K_{i}, M_{i})$$

which finally gives us

$$O_{i} = \sigma(R_{i}) \frac{N_{i}}{D_{i}}$$


key: _node_content
string_value:
{"id_": "f6960842-b042-41f2-8b4a-cad62a1eb8ad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "237d368a-51d7-40b7-8218-61fa59853413", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3eb0f280-ddba-419b-b005-8b09dfb4793d", "node_type": "1", "metadata": {}, "hash": "d51dd502d58096dfb044b6e88d1e14bd808ca8f17971a1d98c9d9f615b8069da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DBRX

## Overview

DBRX is a [transformer-based](https://www.isattentionallyouneed.com/) decoder-only large language model (LLM) that was trained using next-token prediction.
It uses a *fine-grained* mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input.
It was pre-trained on 12T tokens of text and code data.
Compared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2.
This provides 65x more possible combinations of experts and we found that this improves model quality.
DBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA).
It is a BPE based model and uses the GPT-4 tokenizer as described in the [tiktoken](https://github.com/openai/tiktoken) repository.
We made these choices based on exhaustive evaluation and scaling experiments.

DBRX was pretrained on 12T tokens of carefully curated data and a maximum context length of 32K tokens.
We estimate that this data is at least 2x better token-for-token than the data we used to pretrain the MPT family of models.
This new dataset was developed using the full suite of Databricks tools, including Apache Spark™ and Databricks notebooks for data processing, and Unity Catalog for data management and governance.
We used curriculum learning for pretraining, changing the data mix during training in ways we found to substantially improve model quality.


More detailed information about DBRX Instruct and DBRX Base can be found in our [technical blog post](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm).

This model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.

## Usage Examples

The `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.

```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

If you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)


key: _node_content
string_value:
{"id_": "3eb0f280-ddba-419b-b005-8b09dfb4793d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "237d368a-51d7-40b7-8218-61fa59853413", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6960842-b042-41f2-8b4a-cad62a1eb8ad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "b149284a770cf4b201410c88b19b87948d5b7cafa831b00452fb9115c5555418", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7bcca2b-f562-4bc2-a1b1-ab16d0c973d1", "node_type": "1", "metadata": {}, "hash": "5cb61f56f7d938e69d7b06c63e1ccdd68aac00938c7f9bf61ff3644798933570", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1768, "end_char_idx": 4634, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [eitan-turok](https://huggingface.co/eitanturok) and [abhi-db](https://huggingface.co/abhi-db). The original code can be found [here](https://github.com/databricks/dbrx-instruct), though this may not be up to date.

## Usage Examples

The `generate()` method can be used to generate text using DBRX. You can generate using the standard attention implementation, flash-attention, and the PyTorch scaled dot product attention. The last two attention implementations give speed ups.

```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

If you have flash-attention installed (`pip install flash-attn`), it is possible to generate faster. (The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="flash_attention_2",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

You can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="sdpa",
    )

input_text = "What does it take to build a great LLM?"


key: _node_content
string_value:
{"id_": "c7bcca2b-f562-4bc2-a1b1-ab16d0c973d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "237d368a-51d7-40b7-8218-61fa59853413", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "cbeefe8faa3f53bbc16032eaa766d728570874f11046f75511c923cc57c3c9f2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3eb0f280-ddba-419b-b005-8b09dfb4793d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dbrx", "title": "DBRX", "tokens": 1388, "retrieve_doc": true, "source": "transformers"}, "hash": "a519ffefa34499c14e1d1305f0da40d6b1e718ce83f6187dd854c212607c5772", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3068, "end_char_idx": 5086, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
(The HuggingFace documentation for flash-attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="flash_attention_2",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

You can also generate faster using the PyTorch scaled dot product attention. (The HuggingFace documentation for scaled dot product attention can be found [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention).)
```python
from transformers import DbrxForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("databricks/dbrx-instruct", token="YOUR_HF_TOKEN")
model = DbrxForCausalLM.from_pretrained(
    "databricks/dbrx-instruct",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    token="YOUR_HF_TOKEN",
    attn_implementation="sdpa",
    )

input_text = "What does it take to build a great LLM?"
messages = [{"role": "user", "content": input_text}]
input_ids = tokenizer.apply_chat_template(messages, return_dict=True, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))
```

## DbrxConfig

[[autodoc]] DbrxConfig


## DbrxModel

[[autodoc]] DbrxModel
    - forward


## DbrxForCausalLM

[[autodoc]] DbrxForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "2aa0d97c-cb0b-47b9-bb6f-db699e7d4bdc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "750a8169-a7ee-4dae-aa53-3bba533805e8", "node_type": "1", "metadata": {}, "hash": "8e8eddd5f7ab2d7b9b24912405efe684b6db71a6036a80028ecfdfe42a89158f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ByT5

## Overview

The ByT5 model was presented in [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626) by Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir
Kale, Adam Roberts, Colin Raffel.

The abstract from the paper is the following:

*Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.
Encoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from
the model. Token-free models that instead operate directly on raw text (bytes or characters) have many benefits: they
can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by
removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token
sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of
operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with
minimal modifications to process byte sequences. We carefully characterize the trade-offs in terms of parameter count,
training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level
counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on
tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of
pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our
experiments.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://github.com/google-research/byt5).

<Tip>

ByT5's architecture is based on the T5v1.1 model, refer to [T5v1.1's documentation page](t5v1.1) for the API reference. They
only differ in how inputs should be prepared for the model, see the code examples below.

</Tip>

Since ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task
fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.


## Usage example

ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:

```python
>>> from transformers import T5ForConditionalGeneration
>>> import torch

>>> model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")

>>> num_special_tokens = 3
>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.
>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.

>>> input_ids = torch.tensor([list("Life is like a box of chocolates.".encode("utf-8"))]) + num_special_tokens

>>> labels = torch.tensor([list("La vie est comme une boîte de chocolat.


key: _node_content
string_value:
{"id_": "750a8169-a7ee-4dae-aa53-3bba533805e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2aa0d97c-cb0b-47b9-bb6f-db699e7d4bdc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "7aa1bdd2b7ad8a226274f3e143a90d7a8458be97857aa9e1089e40d3429b0493", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7458ae8-c284-450e-b0d5-57cb8253ff81", "node_type": "1", "metadata": {}, "hash": "4937b9471477d1fdadf8a629cdcdff26318a5c7b76842911025759b3e2d6bc13", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2320, "end_char_idx": 4950, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage example

ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:

```python
>>> from transformers import T5ForConditionalGeneration
>>> import torch

>>> model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")

>>> num_special_tokens = 3
>>> # Model has 3 special tokens which take up the input ids 0,1,2 of ByT5.
>>> # => Need to shift utf-8 character encodings by 3 before passing ids to model.

>>> input_ids = torch.tensor([list("Life is like a box of chocolates.".encode("utf-8"))]) + num_special_tokens

>>> labels = torch.tensor([list("La vie est comme une boîte de chocolat.".encode("utf-8"))]) + num_special_tokens

>>> loss = model(input_ids, labels=labels).loss
>>> loss.item()
2.66
```

For batched inference and training it is however recommended to make use of the tokenizer:

```python
>>> from transformers import T5ForConditionalGeneration, AutoTokenizer

>>> model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")
>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-small")

>>> model_inputs = tokenizer(
...     ["Life is like a box of chocolates.", "Today is Monday."], padding="longest", return_tensors="pt"
... )
>>> labels_dict = tokenizer(
...     ["La vie est comme une boîte de chocolat.", "Aujourd'hui c'est lundi."], padding="longest", return_tensors="pt"
... )
>>> labels = labels_dict.input_ids

>>> loss = model(**model_inputs, labels=labels).loss
>>> loss.item()
17.9
```

Similar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, 
since the model works directly on characters, the pretraining task is a bit 
different. Let's corrupt some characters of the 
input sentence `"The dog chases a ball in the park."` and ask ByT5 to predict them 
for us.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/byt5-base")

>>> input_ids_prompt = "The dog chases a ball in the park."
>>> input_ids = tokenizer(input_ids_prompt).input_ids

>>> # Note that we cannot add "{extra_id_...}" to the string directly
>>> # as the Byte tokenizer would incorrectly merge the tokens
>>> # For ByT5, we need to work directly on the character level
>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead
>>> # uses final utf character ids.
>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.
>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.
>>> # => mask to "The dog [258]a ball [257]park."


key: _node_content
string_value:
{"id_": "b7458ae8-c284-450e-b0d5-57cb8253ff81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "750a8169-a7ee-4dae-aa53-3bba533805e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "20100c34e76cea0c27c1ed3eef40ea016b61ed4a6e8aba5fc5225f852104060d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a5c0962-7232-4448-b98c-e9ce6065b0b0", "node_type": "1", "metadata": {}, "hash": "063d4e0a601f19634d41e30172f0792ddfc1faa4844db2875f1eb8450258025d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3469, "end_char_idx": 5319, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"], padding="longest", return_tensors="pt"
... )
>>> labels_dict = tokenizer(
...     ["La vie est comme une boîte de chocolat.", "Aujourd'hui c'est lundi."], padding="longest", return_tensors="pt"
... )
>>> labels = labels_dict.input_ids

>>> loss = model(**model_inputs, labels=labels).loss
>>> loss.item()
17.9
```

Similar to [T5](t5), ByT5 was trained on the span-mask denoising task. However, 
since the model works directly on characters, the pretraining task is a bit 
different. Let's corrupt some characters of the 
input sentence `"The dog chases a ball in the park."` and ask ByT5 to predict them 
for us.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/byt5-base")

>>> input_ids_prompt = "The dog chases a ball in the park."
>>> input_ids = tokenizer(input_ids_prompt).input_ids

>>> # Note that we cannot add "{extra_id_...}" to the string directly
>>> # as the Byte tokenizer would incorrectly merge the tokens
>>> # For ByT5, we need to work directly on the character level
>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead
>>> # uses final utf character ids.
>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.
>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.
>>> # => mask to "The dog [258]a ball [257]park."

>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])
>>> input_ids
tensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])

>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`.


key: _node_content
string_value:
{"id_": "0a5c0962-7232-4448-b98c-e9ce6065b0b0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7458ae8-c284-450e-b0d5-57cb8253ff81", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "89dac8d6a41dacc28ed77b4207dbef97a5b59faf5ed57a834743aca95780c040", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "73054ac0-4c0d-43dd-90f9-152f45b472c2", "node_type": "1", "metadata": {}, "hash": "16f949deb325dceb8c3100c6f57c1d74538b8b4a93110c00fd716dee66d4c944", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4046, "end_char_idx": 6522, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"` and ask ByT5 to predict them 
for us.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("google/byt5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/byt5-base")

>>> input_ids_prompt = "The dog chases a ball in the park."
>>> input_ids = tokenizer(input_ids_prompt).input_ids

>>> # Note that we cannot add "{extra_id_...}" to the string directly
>>> # as the Byte tokenizer would incorrectly merge the tokens
>>> # For ByT5, we need to work directly on the character level
>>> # Contrary to T5, ByT5 does not use sentinel tokens for masking, but instead
>>> # uses final utf character ids.
>>> # UTF-8 is represented by 8 bits and ByT5 has 3 special tokens.
>>> # => There are 2**8+2 = 259 input ids and mask tokens count down from index 258.
>>> # => mask to "The dog [258]a ball [257]park."

>>> input_ids = torch.tensor([input_ids[:8] + [258] + input_ids[14:21] + [257] + input_ids[28:]])
>>> input_ids
tensor([[ 87, 107, 104,  35, 103, 114, 106,  35, 258,  35, 100,  35, 101, 100, 111, 111, 257,  35, 115, 100, 117, 110,  49,   1]])

>>> # ByT5 produces only one char at a time so we need to produce many more output characters here -> set `max_length=100`.
>>> output_ids = model.generate(input_ids, max_length=100)[0].tolist()
>>> output_ids
[0, 258, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118, 257,  35, 108, 113,  35, 119, 107, 104,  35, 103, 108, 118, 102, 114, 256, 108, 113,  35, 119, 107, 104, 35, 115, 100, 117, 110,  49,  35,  87, 107, 104,  35, 103, 114, 106, 35, 108, 118,  35, 119, 107, 104,  35, 114, 113, 104,  35, 122, 107, 114,  35, 103, 114, 104, 118,  35, 100,  35, 101, 100, 111, 111,  35, 108, 113, 255,  35, 108, 113,  35, 119, 107, 104,  35, 115, 100, 117, 110,  49]

>>> # ^- Note how 258 descends to 257, 256, 255

>>> # Now we need to split on the sentinel tokens, let's write a short loop for this
>>> output_ids_list = []
>>> start_token = 0
>>> sentinel_token = 258
>>> while sentinel_token in output_ids:
...     split_idx = output_ids.index(sentinel_token)
...     output_ids_list.append(output_ids[start_token:split_idx])
...     start_token = split_idx
...     sentinel_token -= 1

>>> output_ids_list.append(output_ids[start_token:])
>>> output_string = tokenizer.batch_decode(output_ids_list)
>>> output_string
['<pad>', 'is the one who does', ' in the disco', 'in the park.


key: _node_content
string_value:
{"id_": "73054ac0-4c0d-43dd-90f9-152f45b472c2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8472fab1-94d8-4c02-981c-9a672cc56d3b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "5bec4a969d99f261bb697a9e1acac237a8aea813db6eaee65bef496d484d8428", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a5c0962-7232-4448-b98c-e9ce6065b0b0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/byt5", "title": "ByT5", "tokens": 2066, "retrieve_doc": true, "source": "transformers"}, "hash": "764f4e4d4902264f1622523d98e7dbec19c3e6886060e4dfc68b7dc467ee6095", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6523, "end_char_idx": 6669, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The dog is the one who does a ball in', ' in the park.']
```


## ByT5Tokenizer

[[autodoc]] ByT5Tokenizer

See [`ByT5Tokenizer`] for all details.


key: _node_content
string_value:
{"id_": "6fdf25bc-a77e-47f7-9b06-b86bf613626b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5d6f1937-bfa4-470e-ae37-a6c4d2c27d32", "node_type": "1", "metadata": {}, "hash": "0b4d60cc3a1f981cf028354a64d30a7f2dcefa6502239ed6ef88f9c396d2b289", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2863, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Qwen2_VL


## Overview

The [Qwen2_VL](https://qwenlm.github.io/blog/qwen2-vl/) is a major update to our [Qwen-VL](https://arxiv.org/pdf/2308.12966) model from the Qwen team. 

The abstract from the blog is the following:

*This blog introduces Qwen2-VL, an advanced version of the Qwen-VL model that has undergone significant enhancements over the past year. Key improvements include enhanced image comprehension, advanced video understanding, integrated visual agent functionality, and expanded multilingual support. The model architecture has been optimized for handling arbitrary image resolutions through Naive Dynamic Resolution support and utilizes Multimodal Rotary Position Embedding (M-ROPE) to effectively process both 1D textual and multi-dimensional visual data. This updated model demonstrates competitive performance against leading AI systems like GPT-4o and Claude 3.5 Sonnet in vision-related tasks and ranks highly among open-source models in text capabilities. These advancements make Qwen2-VL a versatile tool for various applications requiring robust multimodal processing and reasoning abilities.*


## Usage example

### Single Media inference

The model can accept both images and videos as input. Here's an example code for inference.

```python

from PIL import Image
import requests
import torch
from torchvision import io
from typing import Dict
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor

# Load the model in half-precision on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", device_map="auto")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

# Image
url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
image = Image.open(requests.get(url, stream=True).raw)

conversation = [
    {
        "role":"user",
        "content":[
            {
                "type":"image",
            },
            {
                "type":"text",
                "text":"Describe this image."
            }
        ]
    }
]


# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\n<|im_start|>assistant\n'

inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")
inputs = inputs.to('cuda')

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)


key: _node_content
string_value:
{"id_": "5d6f1937-bfa4-470e-ae37-a6c4d2c27d32", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6fdf25bc-a77e-47f7-9b06-b86bf613626b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "1e1d41d4a3cff7f045dfb9bc470dfadffcab29f12312e33de07bf216a98868bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e70d5322-d9ed-4e3a-b245-8c1c82decfc6", "node_type": "1", "metadata": {}, "hash": "923b082cb31373e304c72081acff7cf8d55adcf364bf49cd84b977648dd607d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2100, "end_char_idx": 4917, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe this image.<|im_end|>\n<|im_start|>assistant\n'

inputs = processor(text=[text_prompt], images=[image], padding=True, return_tensors="pt")
inputs = inputs.to('cuda')

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)



# Video
def fetch_video(ele: Dict, nframe_factor=2):
    if isinstance(ele['video'], str):
        def round_by_factor(number: int, factor: int) -> int:
            return round(number / factor) * factor

        video = ele["video"]
        if video.startswith("file://"):
            video = video[7:]

        video, _, info = io.read_video(
            video,
            start_pts=ele.get("video_start", 0.0),
            end_pts=ele.get("video_end", None),
            pts_unit="sec",
            output_format="TCHW",
        )
        assert not ("fps" in ele and "nframes" in ele), "Only accept either `fps` or `nframes`"
        if "nframes" in ele:
            nframes = round_by_factor(ele["nframes"], nframe_factor)
        else:
            fps = ele.get("fps", 1.0)
            nframes = round_by_factor(video.size(0) / info["video_fps"] * fps, nframe_factor)
        idx = torch.linspace(0, video.size(0) - 1, nframes, dtype=torch.int64)
        return video[idx]

video_info = {"type": "video", "video": "/path/to/video.mp4", "fps": 1.0}
video = fetch_video(video_info)
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video"},
            {"type": "text", "text": "What happened in the video?"},
        ],
    }
]

# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|video_pad|><|vision_end|>What happened in the video?<|im_end|>\n<|im_start|>assistant\n'

inputs = processor(text=[text_prompt], videos=[video], padding=True, return_tensors="pt")
inputs = inputs.to('cuda')

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)

```


key: _node_content
string_value:
{"id_": "e70d5322-d9ed-4e3a-b245-8c1c82decfc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5d6f1937-bfa4-470e-ae37-a6c4d2c27d32", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "957ffcd5bc726e51a1e809dde0eeb28770d08eee0a151a02cede633d8adb39bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f1a8b286-0d7a-4e7b-841e-e324e0b20c97", "node_type": "1", "metadata": {}, "hash": "765584f10dd1fc2f119a4cc901eb46d8ec82eaa364f2d63e92e55595a2d1b5d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4920, "end_char_idx": 7816, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Batch Mixed Media Inference

The model can batch inputs composed of mixed samples of various types such as images, videos, and text. Here is an example.

```python

image1 = Image.open("/path/to/image1.jpg")
image2 = Image.open("/path/to/image2.jpg")
image3 = Image.open("/path/to/image3.jpg")
image4 = Image.open("/path/to/image4.jpg")
image5 = Image.open("/path/to/image5.jpg")
video = fetch_video({
    "type": "video",
    "video": "/path/to/video.mp4",
    "fps": 1.0
})

# Conversation for the first image
conversation1 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this image."}
        ]
    }
]

# Conversation with two images
conversation2 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "image"},
            {"type": "text", "text": "What is written in the pictures?"}
        ]
    }
]

# Conversation with pure text
conversation3 = [
    {
        "role": "user",
        "content": "who are you?"
    }
]


# Conversation with mixed midia
conversation4 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "image"},
            {"type": "video"},
            {"type": "text", "text": "What are the common elements in these medias?"},
        ],
    }
]

conversations = [conversation1, conversation2, conversation3, conversation4]
# Preparation for batch inference
texts = [processor.apply_chat_template(msg, add_generation_prompt=True) for msg in conversations]
inputs = processor(
    text=texts,
    images=[image1, image2, image3, image4, image5],
    videos=[video],
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to('cuda')

# Batch Inference
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)
```

### Usage Tips

#### Image Resolution for performance boost

The model supports a wide range of resolution inputs. By default, it uses the native resolution for input, but higher resolutions can enhance performance at the cost of more computation. Users can set the minimum and maximum number of pixels to achieve an optimal configuration for their needs.

```python

min_pixels = 224*224
max_pixels = 2048*2048
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

```



#### Multiple Image Inputs

By default, images and video content are directly included in the conversation. When handling multiple images, it's helpful to add labels to the images and videos for better reference. Users can control this behavior with the following settings:


key: _node_content
string_value:
{"id_": "f1a8b286-0d7a-4e7b-841e-e324e0b20c97", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e70d5322-d9ed-4e3a-b245-8c1c82decfc6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "5d3acd7cad98a4b656a831c0237a9433bcf4e0a28c46ae1e4a11ed59b31ec64e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd2a7bef-e0e1-4235-aa1a-5e8adf2f675a", "node_type": "1", "metadata": {}, "hash": "d36aa92fc769ee10e69019965ad0ec3d2d7600d37e14032eb8d02181e3fc6a26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7543, "end_char_idx": 9653, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Multiple Image Inputs

By default, images and video content are directly included in the conversation. When handling multiple images, it's helpful to add labels to the images and videos for better reference. Users can control this behavior with the following settings:



```python

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"}, 
            {"type": "text", "text": "Hello, how are you?"}
        ]
    },
    {
        "role": "assistant",
        "content": "I'm doing well, thank you for asking. How can I assist you today?"
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Can you describe these images and video?"}, 
            {"type": "image"}, 
            {"type": "image"}, 
            {"type": "video"}, 
            {"type": "text", "text": "These are from my vacation."}
        ]
    },
    {
        "role": "assistant",
        "content": "I'd be happy to describe the images and video for you. Could you please provide more context about your vacation?"
    },
    {
        "role": "user",
        "content": "It was a trip to the mountains. Can you see the details in the images and video?"
    }
]

# default:
prompt_without_id = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Hello, how are you?<|im_end|>\n<|im_start|>assistant\nI'm doing well, thank you for asking. How can I assist you today?<|im_end|>\n<|im_start|>user\nCan you describe these images and video?<|vision_start|><|image_pad|><|vision_end|><|vision_start|><|image_pad|><|vision_end|><|vision_start|><|video_pad|><|vision_end|>These are from my vacation.<|im_end|>\n<|im_start|>assistant\nI'd be happy to describe the images and video for you. Could you please provide more context about your vacation?<|im_end|>\n<|im_start|>user\nIt was a trip to the mountains. Can you see the details in the images and video?<|im_end|>\n<|im_start|>assistant\n'


key: _node_content
string_value:
{"id_": "dd2a7bef-e0e1-4235-aa1a-5e8adf2f675a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93cebb7f-4797-48ef-92a3-21f9fb96ebd7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "2fc8e4b0b8ac9b8d0ca57e6642ffb81888e062f0d3c11a45f4c62dcb738337b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f1a8b286-0d7a-4e7b-841e-e324e0b20c97", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_vl", "title": "Qwen2_VL", "tokens": 3135, "retrieve_doc": true, "source": "transformers"}, "hash": "071fef8913cd1a27861d1ab971984659a9a07990a38c9a267b50737bc4bd06cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9656, "end_char_idx": 11794, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# add ids
prompt_with_id = processor.apply_chat_template(conversation, add_generation_prompt=True, add_vision_id=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nPicture 1: <|vision_start|><|image_pad|><|vision_end|>Hello, how are you?<|im_end|>\n<|im_start|>assistant\nI'm doing well, thank you for asking. How can I assist you today?<|im_end|>\n<|im_start|>user\nCan you describe these images and video?Picture 2: <|vision_start|><|image_pad|><|vision_end|>Picture 3: <|vision_start|><|image_pad|><|vision_end|>Video 1: <|vision_start|><|video_pad|><|vision_end|>These are from my vacation.<|im_end|>\n<|im_start|>assistant\nI'd be happy to describe the images and video for you. Could you please provide more context about your vacation?<|im_end|>\n<|im_start|>user\nIt was a trip to the mountains. Can you see the details in the images and video?<|im_end|>\n<|im_start|>assistant\n'

```

#### Flash-Attention 2 to speed up generation

First, make sure to install the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

Also, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.

To load and run a model using Flash Attention-2, simply add `attn_implementation="flash_attention_2"` when loading the model as follows:

```python
from transformers import Qwen2VLForConditionalGeneration

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", 
    torch_dtype=torch.bfloat16, 
    attn_implementation="flash_attention_2",
)
```


## Qwen2VLConfig

[[autodoc]] Qwen2VLConfig

## Qwen2VLImageProcessor

[[autodoc]] Qwen2VLImageProcessor
    - preprocess

## Qwen2VLProcessor

[[autodoc]] Qwen2VLProcessor

## Qwen2VLModel

[[autodoc]] Qwen2VLModel
    - forward

## Qwen2VLForConditionalGeneration

[[autodoc]] Qwen2VLForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "cd539eca-e2a4-4603-bde1-b2c6acb46e5a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "842e396a-f485-43bc-91d1-d0d43c66bfd1", "node_type": "1", "metadata": {}, "hash": "6a9731a06952e7a5b0573498451d0fdf908e2a71fffc0d342a0e0be5c8ecf324", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SuperPoint

## Overview

The SuperPoint model was proposed
in [SuperPoint: Self-Supervised Interest Point Detection and Description](https://arxiv.org/abs/1712.07629) by Daniel
DeTone, Tomasz Malisiewicz and Andrew Rabinovich.

This model is the result of a self-supervised training of a fully-convolutional network for interest point detection and
description. The model is able to detect interest points that are repeatable under homographic transformations and
provide a descriptor for each point. The use of the model in its own is limited, but it can be used as a feature
extractor for other tasks such as homography estimation, image matching, etc.

The abstract from the paper is the following:

*This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a
large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our
fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and
associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography
approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
when compared to LIFT, SIFT and ORB.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png"
alt="drawing" width="500"/>

<small> SuperPoint overview. Taken from the <a href="https://arxiv.org/abs/1712.07629v4">original paper.</a> </small>

## Usage tips

Here is a quick example of using the model to detect interest points in an image:

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
```

The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).

You can also feed multiple images to the model.


key: _node_content
string_value:
{"id_": "842e396a-f485-43bc-91d1-d0d43c66bfd1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cd539eca-e2a4-4603-bde1-b2c6acb46e5a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "4f6e5e9edf3cf5683a7d0d11efbc62f2bb2092fdb8c61eff3c9efcd337c49dc1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44071b74-b6dd-4219-8101-3a709ad6a705", "node_type": "1", "metadata": {}, "hash": "09da46a3b3df86abdc55d0ec6994a73466c32e30f9c1281aa502b712e3117a07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1100, "end_char_idx": 4342, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We introduce Homographic Adaptation, a multi-scale, multi-homography
approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g.,
synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able
to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other
traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches
when compared to LIFT, SIFT and ORB.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/superpoint_architecture.png"
alt="drawing" width="500"/>

<small> SuperPoint overview. Taken from the <a href="https://arxiv.org/abs/1712.07629v4">original paper.</a> </small>

## Usage tips

Here is a quick example of using the model to detect interest points in an image:

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
```

The outputs contain the list of keypoint coordinates with their respective score and description (a 256-long vector).

You can also feed multiple images to the model. Due to the nature of SuperPoint, to output a dynamic number of keypoints,
you will need to use the mask attribute to retrieve the respective information :

```python
from transformers import AutoImageProcessor, SuperPointForKeypointDetection
import torch
from PIL import Image
import requests

url_image_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
url_image_2 = "http://images.cocodataset.org/test-stuff2017/000000000568.jpg"
image_2 = Image.open(requests.get(url_image_2, stream=True).raw)

images = [image_1, image_2]

processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint")
model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint")

inputs = processor(images, return_tensors="pt")
outputs = model(**inputs)

for i in range(len(images)):
    image_mask = outputs.mask[i]
    image_indices = torch.nonzero(image_mask).squeeze()
    image_keypoints = outputs.keypoints[i][image_indices]
    image_scores = outputs.scores[i][image_indices]
    image_descriptors = outputs.descriptors[i][image_indices]
```

You can then print the keypoints on the image to visualize the result :
```python
import cv2
for keypoint, score in zip(image_keypoints, image_scores):
    keypoint_x, keypoint_y = int(keypoint[0].item()), int(keypoint[1].item())
    color = tuple([score.item() * 255] * 3)
    image = cv2.circle(image, (keypoint_x, keypoint_y), 2, color)
cv2.imwrite("output_image.png", image)
```

This model was contributed by [stevenbucaille](https://huggingface.co/stevenbucaille).


key: _node_content
string_value:
{"id_": "44071b74-b6dd-4219-8101-3a709ad6a705", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "70293822-19dc-4610-b50b-9f0c65b05d11", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "55756f26bbf925ee6b445d381583fdc3b7afcc11cf98a7cb570bd3397581014e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "842e396a-f485-43bc-91d1-d0d43c66bfd1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/superpoint", "title": "SuperPoint", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "d1a4e4198be167c360e4faf1b0b0e392f2ba722562e0c9045f0fad1f12a4c36e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4343, "end_char_idx": 5256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/magicleap/SuperPointPretrainedNetwork).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SuperPoint. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- A notebook showcasing inference and visualization with SuperPoint can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SuperPoint/Inference_with_SuperPoint_to_detect_interest_points_in_an_image.ipynb). 🌎

## SuperPointConfig

[[autodoc]] SuperPointConfig

## SuperPointImageProcessor

[[autodoc]] SuperPointImageProcessor

- preprocess

## SuperPointForKeypointDetection

[[autodoc]] SuperPointForKeypointDetection

- forward


key: _node_content
string_value:
{"id_": "44ef598c-79ba-42e1-9ccb-311bc92b3070", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "06c2897d-3181-42d2-ac47-bc64d060f004", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/flan-t5", "title": "FLAN-T5", "tokens": 610, "retrieve_doc": true, "source": "transformers"}, "hash": "052e45c889c784ad2c1f26dba252e6414d6b231f65ccd22d42dad8339a24f72b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1685, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FLAN-T5

## Overview

FLAN-T5 was released in the paper [Scaling Instruction-Finetuned Language Models](https://arxiv.org/pdf/2210.11416.pdf) - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.

One can directly use FLAN-T5 weights without finetuning the model:

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
>>> tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")

>>> inputs = tokenizer("A step by step recipe to make bolognese pasta:", return_tensors="pt")
>>> outputs = model.generate(**inputs)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Pour a cup of bolognese into a large bowl and add the pasta']
```

FLAN-T5 includes the same improvements as T5 version 1.1 (see [here](https://huggingface.co/docs/transformers/model_doc/t5v1.1) for the full details of the model's improvements.)

Google has released the following variants:

- [google/flan-t5-small](https://huggingface.co/google/flan-t5-small)

- [google/flan-t5-base](https://huggingface.co/google/flan-t5-base)

- [google/flan-t5-large](https://huggingface.co/google/flan-t5-large)

- [google/flan-t5-xl](https://huggingface.co/google/flan-t5-xl)

- [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl).

The original checkpoints can be found [here](https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints).

<Tip>

Refer to [T5's documentation page](t5) for all API reference, code examples and notebooks. For more details regarding training and evaluation of the FLAN-T5, refer to the model card.

</Tip>


key: _node_content
string_value:
{"id_": "b2f59fc8-8fd2-47d0-8289-0e21ec2b4d62", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "hash": "bcb5f3c5f15a803e49e6dc0dd3250147113bbf89e466bef5c5560eb852212c90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa9580dc-6f6b-466c-be79-e147d9dc94e5", "node_type": "1", "metadata": {}, "hash": "38303f5ac0f88881c98e64927f80261931afd54ef45b4dae38fd068a8640a888", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2864, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CTRL

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=ctrl">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-ctrl-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/tiny-ctrl">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

CTRL model was proposed in [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus
of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).

The abstract from the paper is the following:

*Large-scale language models show promising text generation capabilities, but users cannot easily control particular
aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,
trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while
providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the
training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data
via model-based source attribution.*

This model was contributed by [keskarnitishr](https://huggingface.co/keskarnitishr). The original code can be found
[here](https://github.com/salesforce/ctrl).

## Usage tips

- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences
  or links to generate coherent text. Refer to the [original implementation](https://github.com/salesforce/ctrl) for
  more information.
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be
  observed in the *run_generation.py* example script.
- The PyTorch models can take the `past_key_values` as input, which is the previously computed key/value attention pairs.
  TensorFlow models accepts `past` as input. Using the `past_key_values` value prevents the model from re-computing
  pre-computed values in the context of text generation. See the [`forward`](model_doc/ctrl#transformers.CTRLModel.forward)
  method for more information on the usage of this argument.


key: _node_content
string_value:
{"id_": "aa9580dc-6f6b-466c-be79-e147d9dc94e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "20533bdf-7520-43fb-9c1c-4ac90808f7d6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "hash": "bcb5f3c5f15a803e49e6dc0dd3250147113bbf89e466bef5c5560eb852212c90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b2f59fc8-8fd2-47d0-8289-0e21ec2b4d62", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ctrl", "title": "CTRL", "tokens": 1027, "retrieve_doc": true, "source": "transformers"}, "hash": "9f1879bacff36180959022e24a0a384264c8e5f42c182c18ed7e96183a7fa3ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2867, "end_char_idx": 3600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## CTRLConfig

[[autodoc]] CTRLConfig

## CTRLTokenizer

[[autodoc]] CTRLTokenizer
    - save_vocabulary

<frameworkcontent>
<pt>

## CTRLModel

[[autodoc]] CTRLModel
    - forward

## CTRLLMHeadModel

[[autodoc]] CTRLLMHeadModel
    - forward

## CTRLForSequenceClassification

[[autodoc]] CTRLForSequenceClassification
    - forward

</pt>
<tf>

## TFCTRLModel

[[autodoc]] TFCTRLModel
    - call

## TFCTRLLMHeadModel

[[autodoc]] TFCTRLLMHeadModel
    - call

## TFCTRLForSequenceClassification

[[autodoc]] TFCTRLForSequenceClassification
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "46a364f3-b8ae-4c56-88a0-89cc519a63c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/git", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "286c0ee5-7d2a-4e6b-8e41-6fed3bf0e2ee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/git", "title": "GIT", "tokens": 888, "retrieve_doc": true, "source": "transformers"}, "hash": "8e695eda445ff55612b51c26331ca234eec8ab873d31219f8b7e7c098841f693", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3136, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GIT

## Overview

The GIT model was proposed in [GIT: A Generative Image-to-text Transformer for Vision and Language](https://arxiv.org/abs/2205.14100) by
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. GIT is a decoder-only Transformer
that leverages [CLIP](clip)'s vision encoder to condition the model on vision inputs besides text. The model obtains state-of-the-art results on
image captioning and visual question answering benchmarks.

The abstract from the paper is the following:

*In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/git_architecture.jpg"
alt="drawing" width="600"/>

<small> GIT architecture. Taken from the <a href="https://arxiv.org/abs/2205.14100" target="_blank">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/GenerativeImage2Text).

## Usage tips

- GIT is implemented in a very similar way to GPT-2, the only difference being that the model is also conditioned on `pixel_values`.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GIT.

- Demo notebooks regarding inference + fine-tuning GIT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GIT).
- See also: [Causal language modeling task guide](../tasks/language_modeling)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.
The resource should ideally demonstrate something new instead of duplicating an existing resource.

## GitVisionConfig

[[autodoc]] GitVisionConfig

## GitVisionModel

[[autodoc]] GitVisionModel
    - forward

## GitConfig

[[autodoc]] GitConfig
    - all

## GitProcessor

[[autodoc]] GitProcessor
    - __call__

## GitModel

[[autodoc]] GitModel
    - forward

## GitForCausalLM

[[autodoc]] GitForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "9b28eec5-8a8e-4a50-a470-459971100f2c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clap", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9f4b4deb-bba4-42c1-9cd2-07f4bdd748fd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clap", "title": "CLAP", "tokens": 857, "retrieve_doc": true, "source": "transformers"}, "hash": "2952311c91ef6ca8ffcf841bfcec17524cfd1ce84e1e603f134c6e8d96ab6dce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLAP

## Overview

The CLAP model was proposed in [Large Scale Contrastive Language-Audio pretraining with
feature fusion and keyword-to-caption augmentation](https://arxiv.org/pdf/2211.06687.pdf) by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.

CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.

The abstract from the paper is the following:

*Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-6*

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ) .
The original code can be found [here](https://github.com/LAION-AI/Clap).

## ClapConfig

[[autodoc]] ClapConfig
    - from_text_audio_configs

## ClapTextConfig

[[autodoc]] ClapTextConfig

## ClapAudioConfig

[[autodoc]] ClapAudioConfig

## ClapFeatureExtractor

[[autodoc]] ClapFeatureExtractor

## ClapProcessor

[[autodoc]] ClapProcessor

## ClapModel

[[autodoc]] ClapModel
    - forward
    - get_text_features
    - get_audio_features

## ClapTextModel

[[autodoc]] ClapTextModel
    - forward

## ClapTextModelWithProjection

[[autodoc]] ClapTextModelWithProjection
    - forward

## ClapAudioModel

[[autodoc]] ClapAudioModel
    - forward

## ClapAudioModelWithProjection

[[autodoc]] ClapAudioModelWithProjection
    - forward


key: _node_content
string_value:
{"id_": "a90763d6-89e5-4dae-a594-d17b30591f2a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "hash": "90cdafd1166bebb76ce42f5770187f53677ad7409847c087ac3f35a265a53405", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "02e171af-0a96-4d54-83d9-bfc5c831dc58", "node_type": "1", "metadata": {}, "hash": "10b1b79716168c7d477c667436e603efe4a3b6755e957eb60bc18aafe6014e1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CLVP

## Overview

The CLVP (Contrastive Language-Voice Pretrained Transformer) model was proposed in [Better speech synthesis through scaling](https://arxiv.org/abs/2305.07243) by James Betker.

The abstract from the paper is the following:

*In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise - an expressive, multi-voice text-to-speech system.*


This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/neonbjb/tortoise-tts).


## Usage tips

1. CLVP is an integral part of the Tortoise TTS model.
2. CLVP can be used to compare different generated speech candidates with the provided text, and the best speech tokens are forwarded to the diffusion model.
3. The use of the [`ClvpModelForConditionalGeneration.generate()`] method is strongly recommended for tortoise usage.
4. Note that the CLVP model expects the audio to be sampled at 22.05 kHz contrary to other audio models which expects 16 kHz. 


## Brief Explanation:

- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.
- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.
- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.
- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. 
- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. 
- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  


Example :

```python
>>> import datasets
>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration

>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).
>>> text = "This is an example text."

>>> ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050))
>>> sample = ds[0]["audio"]

>>> # Define processor and model.
>>> processor = ClvpProcessor.from_pretrained("susnato/clvp_dev")
>>> model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev")

>>> # Generate processor output and model output.
>>> processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt")
>>> generated_output = model.generate(**processor_output)
```


key: _node_content
string_value:
{"id_": "02e171af-0a96-4d54-83d9-bfc5c831dc58", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7189666f-ebfc-4324-bea3-f2880f59cee9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "hash": "90cdafd1166bebb76ce42f5770187f53677ad7409847c087ac3f35a265a53405", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a90763d6-89e5-4dae-a594-d17b30591f2a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/clvp", "title": "CLVP", "tokens": 1096, "retrieve_doc": true, "source": "transformers"}, "hash": "0cd7f60b27916b5d3501ae71dedfd75acc6c8d6d54d515de4a8db1901fdd91a7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1436, "end_char_idx": 3958, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Brief Explanation:

- The [`ClvpTokenizer`] tokenizes the text input, and the [`ClvpFeatureExtractor`] extracts the log mel-spectrogram from the desired audio.
- [`ClvpConditioningEncoder`] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio.
- The [`ClvpForCausalLM`] uses those embeddings to generate multiple speech candidates.
- Each speech candidate is passed through the speech encoder ([`ClvpEncoder`]) which converts them into a vector representation, and the text encoder ([`ClvpEncoder`]) converts the text tokens into the same latent space. 
- At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. 
- [`ClvpModelForConditionalGeneration.generate()`] compresses all of the logic described above into a single method.  


Example :

```python
>>> import datasets
>>> from transformers import ClvpProcessor, ClvpModelForConditionalGeneration

>>> # Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using `datasets` library).
>>> text = "This is an example text."

>>> ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050))
>>> sample = ds[0]["audio"]

>>> # Define processor and model.
>>> processor = ClvpProcessor.from_pretrained("susnato/clvp_dev")
>>> model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev")

>>> # Generate processor output and model output.
>>> processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt")
>>> generated_output = model.generate(**processor_output)
```


## ClvpConfig

[[autodoc]] ClvpConfig
    - from_sub_model_configs

## ClvpEncoderConfig

[[autodoc]] ClvpEncoderConfig

## ClvpDecoderConfig

[[autodoc]] ClvpDecoderConfig

## ClvpTokenizer

[[autodoc]] ClvpTokenizer
    - save_vocabulary

## ClvpFeatureExtractor

[[autodoc]] ClvpFeatureExtractor
    - __call__

## ClvpProcessor

[[autodoc]] ClvpProcessor
    - __call__
    - decode
    - batch_decode

## ClvpModelForConditionalGeneration

[[autodoc]] ClvpModelForConditionalGeneration
    - forward
    - generate
    - get_text_features
    - get_speech_features

## ClvpForCausalLM

[[autodoc]] ClvpForCausalLM

## ClvpModel

[[autodoc]] ClvpModel

## ClvpEncoder

[[autodoc]] ClvpEncoder

## ClvpDecoder

[[autodoc]] ClvpDecoder


key: _node_content
string_value:
{"id_": "0ef6d2b0-4a47-44a0-997b-b35701176cae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3ea0dd8-a6fc-41d4-b197-e63e52f8a08e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/recurrent_gemma", "title": "RecurrentGemma", "tokens": 505, "retrieve_doc": true, "source": "transformers"}, "hash": "baf5f71b6e8653a8aca416eef2e7a45d85b5901d8481ce2cadf41070e00d7585", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RecurrentGemma

## Overview

The Recurrent Gemma model was proposed in [RecurrentGemma: Moving Past Transformers for Efficient Open Language Models](https://storage.googleapis.com/deepmind-media/gemma/recurrentgemma-report.pdf) by the Griffin, RLHF and Gemma Teams of Google.

The abstract from the paper is the following:

*We introduce RecurrentGemma, an open language model which uses Google’s novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.*

Tips:

- The original checkpoints can be converted using the conversion script [`src/transformers/models/recurrent_gemma/convert_recurrent_gemma_weights_to_hf.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/recurrent_gemma/convert_recurrent_gemma_to_hf.py). 

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The original code can be found [here](https://github.com/google-deepmind/recurrentgemma).


## RecurrentGemmaConfig

[[autodoc]] RecurrentGemmaConfig


## RecurrentGemmaModel

[[autodoc]] RecurrentGemmaModel
    - forward

## RecurrentGemmaForCausalLM

[[autodoc]] RecurrentGemmaForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "bbb6d332-7e3d-47f4-9d9b-f0f2c7233293", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "73a381bf-066c-4853-acb1-90fff736433f", "node_type": "1", "metadata": {}, "hash": "7cc0bb5b80177fc43726039da8bbe670c7e2fc7261cb0c4965eaced23a0ba463", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3289, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DETR

## Overview

The DETR model was proposed in [End-to-End Object Detection with Transformers](https://arxiv.org/abs/2005.12872) by
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. DETR
consists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for
object detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use
things like region proposals, non-maximum suppression procedure and anchor generation. Moreover, DETR can also be
naturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.

The abstract from the paper is the following:

*We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the
detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression
procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the
new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via
bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries,
DETR reasons about the relations of the objects and the global image context to directly output the final set of
predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many
other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and
highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily
generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive
baselines.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).

## How DETR works

Here's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:

First, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use
ResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a
tensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone
outputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is
then projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a
`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the
feature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =
`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually
longer than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).

Next, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider
these as image features).


key: _node_content
string_value:
{"id_": "73a381bf-066c-4853-acb1-90fff736433f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bbb6d332-7e3d-47f4-9d9b-f0f2c7233293", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "8133511a188276b66d7ff74afe79471446075eaf74533ef3921005f4f451f77c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6c9dbb1-d626-4305-aecd-7258e302de88", "node_type": "1", "metadata": {}, "hash": "23941821e25ce2b1aa060ae85a540c4ab9aca2358db261fdbc09e5e92424a9fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1778, "end_char_idx": 5025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Moreover, DETR can be easily
generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive
baselines.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/facebookresearch/detr).

## How DETR works

Here's a TLDR explaining how [`~transformers.DetrForObjectDetection`] works:

First, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use
ResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a
tensor of shape `(batch_size, 3, height, width)`, assuming the image has 3 color channels (RGB). The CNN backbone
outputs a new lower-resolution feature map, typically of shape `(batch_size, 2048, height/32, width/32)`. This is
then projected to match the hidden dimension of the Transformer of DETR, which is `256` by default, using a
`nn.Conv2D` layer. So now, we have a tensor of shape `(batch_size, 256, height/32, width/32).` Next, the
feature map is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` =
`(batch_size, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually
longer than usual, but with a smaller `d_model` (which in NLP is typically 768 or higher).

Next, this is sent through the encoder, outputting `encoder_hidden_states` of the same shape (you can consider
these as image features). Next, so-called **object queries** are sent through the decoder. This is a tensor of shape
`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.
These input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to
the encoder, they are added to the input of each attention layer. Each object query will look for a particular object
in the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers
to output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads
are added on top for object detection: a linear layer for classifying each object query into one of the objects or "no
object", and a MLP to predict bounding boxes for each query.

The model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +
bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N
(so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as
bounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find
an optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for
the classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the
bounding boxes) are used to optimize the parameters of the model.

DETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance
segmentation).


key: _node_content
string_value:
{"id_": "f6c9dbb1-d626-4305-aecd-7258e302de88", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "73a381bf-066c-4853-acb1-90fff736433f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "d88b6fe2d8433ef1c414f3be72a0966e68d29a0d4b63b761530a8641d64b65cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "14e7640e-f037-4acd-8345-c0021c5b369a", "node_type": "1", "metadata": {}, "hash": "981778274d11ee1aaa6676a47f03930acc7a7e47bf9b12c3828c00c246a9aa6f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3290, "end_char_idx": 6686, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next, so-called **object queries** are sent through the decoder. This is a tensor of shape
`(batch_size, num_queries, d_model)`, with `num_queries` typically set to 100 and initialized with zeros.
These input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to
the encoder, they are added to the input of each attention layer. Each object query will look for a particular object
in the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers
to output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads
are added on top for object detection: a linear layer for classifying each object query into one of the objects or "no
object", and a MLP to predict bounding boxes for each query.

The model is trained using a **bipartite matching loss**: so what we actually do is compare the predicted classes +
bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N
(so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as
bounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to find
an optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for
the classes) and a linear combination of the L1 and [generalized IoU loss](https://giou.stanford.edu/) (for the
bounding boxes) are used to optimize the parameters of the model.

DETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance
segmentation). [`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of
[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,
where one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both
"things" (instances) and "stuff" (background things like trees, roads, sky), then freeze all the weights and train only
the mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is
required for the training to be possible, since the Hungarian matching is computed using distances between boxes.

## Usage tips

- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum
  number of objects that can be detected in a single image, and is set to 100 by default (see parameter
  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the
  authors used 100, while the maximum number of objects in a COCO image is ~70).
- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,
  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.
- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting
  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned
  absolute position embeddings. By default, the parameter `position_embedding_type` of
  [`~transformers.DetrConfig`] is set to `"sine"`.


key: _node_content
string_value:
{"id_": "14e7640e-f037-4acd-8345-c0021c5b369a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6c9dbb1-d626-4305-aecd-7258e302de88", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "9a4bdd12fe0a50f218bc637639c1fc6c845308868ac1bf2bf8c9fc83828da0d6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90e3a577-58e4-494d-9255-fc3a7e8e30cd", "node_type": "1", "metadata": {}, "hash": "e8abbb78a57232433b5617e627f1746cdae35cb8de3362664cd2bfdd92f3f0bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5026, "end_char_idx": 8320, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`~transformers.DetrForSegmentation`] adds a segmentation mask head on top of
[`~transformers.DetrForObjectDetection`]. The mask head can be trained either jointly, or in a two steps process,
where one first trains a [`~transformers.DetrForObjectDetection`] model to detect bounding boxes around both
"things" (instances) and "stuff" (background things like trees, roads, sky), then freeze all the weights and train only
the mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is
required for the training to be possible, since the Hungarian matching is computed using distances between boxes.

## Usage tips

- DETR uses so-called **object queries** to detect objects in an image. The number of queries determines the maximum
  number of objects that can be detected in a single image, and is set to 100 by default (see parameter
  `num_queries` of [`~transformers.DetrConfig`]). Note that it's good to have some slack (in COCO, the
  authors used 100, while the maximum number of objects in a COCO image is ~70).
- The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,
  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.
- DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting
  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned
  absolute position embeddings. By default, the parameter `position_embedding_type` of
  [`~transformers.DetrConfig`] is set to `"sine"`.
- During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help
  the model output the correct number of objects of each class. If you set the parameter `auxiliary_loss` of
  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses
  are added after each decoder layer (with the FFNs sharing parameters).
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be
  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).
- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with
  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).
  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of
  [`~transformers.DetrConfig`] to `"tf_mobilenetv3_small_075"`, and then initializing the model with that
  config.
- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is
  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at
  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800.


key: _node_content
string_value:
{"id_": "90e3a577-58e4-494d-9255-fc3a7e8e30cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "14e7640e-f037-4acd-8345-c0021c5b369a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "dbbc7498d8e892fbd60aeb558160e59efd14e2771004e528688d6e0f1682e134", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3", "node_type": "1", "metadata": {}, "hash": "6659784809a4f14b87e33413e6ae2022780aaaae3758eabb9f9574ab3e1e0b60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6869, "end_char_idx": 10172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you set the parameter `auxiliary_loss` of
  [`~transformers.DetrConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses
  are added after each decoder layer (with the FFNs sharing parameters).
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  _num_boxes_ variable in the _DetrLoss_ class of _modeling_detr.py_. When training on multiple nodes, this should be
  set to the average number of target boxes across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L227-L232).
- [`~transformers.DetrForObjectDetection`] and [`~transformers.DetrForSegmentation`] can be initialized with
  any convolutional backbone available in the [timm library](https://github.com/rwightman/pytorch-image-models).
  Initializing with a MobileNet backbone for example can be done by setting the `backbone` attribute of
  [`~transformers.DetrConfig`] to `"tf_mobilenetv3_small_075"`, and then initializing the model with that
  config.
- DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is
  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at
  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800. One can use
  [`~transformers.DetrImageProcessor`] to prepare images (and optional annotations in COCO format) for the
  model. Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the
  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.
  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using
  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].
- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.
  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.

There are three ways to instantiate a DETR model (depending on what you prefer):

Option 1: Instantiate DETR with pre-trained weights for entire model
```py
>>> from transformers import DetrForObjectDetection

>>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
```

Option 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone
```py
>>> from transformers import DetrConfig, DetrForObjectDetection

>>> config = DetrConfig()
>>> model = DetrForObjectDetection(config)
```
Option 3: Instantiate DETR with randomly initialized weights for backbone + Transformer
```py
>>> config = DetrConfig(use_pretrained_backbone=False)
>>> model = DetrForObjectDetection(config)
```

As a summary, consider the following table:

| Task | Object detection | Instance segmentation | Panoptic segmentation |
|------|------------------|-----------------------|-----------------------|
| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e.


key: _node_content
string_value:
{"id_": "50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90e3a577-58e4-494d-9255-fc3a7e8e30cd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "9ae0219487690477498644579e5a3168eea572fd4b45b24f15445e6c7eb37cdb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d8e564f-0c72-411d-8795-0b04d5ec63f6", "node_type": "1", "metadata": {}, "hash": "c095f323596f810559cad014578152e749735cd92decc98e9f4cd0d7b7291f54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8449, "end_char_idx": 11155, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the
  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.
  Alternatively, one can also define a custom `collate_fn` in order to batch images together, using
  [`~transformers.DetrImageProcessor.pad_and_create_pixel_mask`].
- The size of the images will determine the amount of memory being used, and will thus determine the `batch_size`.
  It is advised to use a batch size of 2 per GPU. See [this Github thread](https://github.com/facebookresearch/detr/issues/150) for more info.

There are three ways to instantiate a DETR model (depending on what you prefer):

Option 1: Instantiate DETR with pre-trained weights for entire model
```py
>>> from transformers import DetrForObjectDetection

>>> model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
```

Option 2: Instantiate DETR with randomly initialized weights for Transformer, but pre-trained weights for backbone
```py
>>> from transformers import DetrConfig, DetrForObjectDetection

>>> config = DetrConfig()
>>> model = DetrForObjectDetection(config)
```
Option 3: Instantiate DETR with randomly initialized weights for backbone + Transformer
```py
>>> config = DetrConfig(use_pretrained_backbone=False)
>>> model = DetrForObjectDetection(config)
```

As a summary, consider the following table:

| Task | Object detection | Instance segmentation | Panoptic segmentation |
|------|------------------|-----------------------|-----------------------|
| **Description** | Predicting bounding boxes and class labels around objects in an image | Predicting masks around objects (i.e. instances) in an image | Predicting masks around both objects (i.e. instances) as well as "stuff" (i.e. background things like trees and roads) in an image |
| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |
| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |
| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |
| **Postprocessing** (i.e.


key: _node_content
string_value:
{"id_": "1d8e564f-0c72-411d-8795-0b04d5ec63f6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50ff22a9-9db6-45bd-90e2-3f6e5c9eaae3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "21c43483412201cf1ef457d256d1ab7ab66cdd8e06071d2da8fda6e4c6fec1a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37b1ae69-2a82-4a99-a7a3-9eabdc34d35f", "node_type": "1", "metadata": {}, "hash": "160e1d9ce59d6d183ea4fafeaca9384dca7efdad9bec24cfb6e4c439882240ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10173, "end_char_idx": 13285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
instances) in an image | Predicting masks around both objects (i.e. instances) as well as "stuff" (i.e. background things like trees and roads) in an image |
| **Model** | [`~transformers.DetrForObjectDetection`] | [`~transformers.DetrForSegmentation`] | [`~transformers.DetrForSegmentation`] |
| **Example dataset** | COCO detection | COCO detection, COCO panoptic | COCO panoptic  |                                                                        |
| **Format of annotations to provide to**  [`~transformers.DetrImageProcessor`] | {'image_id': `int`, 'annotations': `List[Dict]`} each Dict being a COCO object annotation  | {'image_id': `int`, 'annotations': `List[Dict]`}  (in case of COCO detection) or {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} (in case of COCO panoptic) | {'file_name': `str`, 'image_id': `int`, 'segments_info': `List[Dict]`} and masks_path (path to directory containing PNG files of the masks) |
| **Postprocessing** (i.e. converting the output of the model to Pascal VOC format) | [`~transformers.DetrImageProcessor.post_process`] | [`~transformers.DetrImageProcessor.post_process_segmentation`] | [`~transformers.DetrImageProcessor.post_process_segmentation`], [`~transformers.DetrImageProcessor.post_process_panoptic`] |
| **evaluators** | `CocoEvaluator` with `iou_types="bbox"` | `CocoEvaluator` with `iou_types="bbox"` or `"segm"` | `CocoEvaluator` with `iou_tupes="bbox"` or `"segm"`, `PanopticEvaluator` |

In short, one should prepare the data either in COCO detection or COCO panoptic format, then use
[`~transformers.DetrImageProcessor`] to create `pixel_values`, `pixel_mask` and optional
`labels`, which can then be used to train (or fine-tune) a model. For evaluation, one should first convert the
outputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can
be be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like
mean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DETR.

<PipelineTag pipeline="object-detection"/>

- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).
- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it!


key: _node_content
string_value:
{"id_": "37b1ae69-2a82-4a99-a7a3-9eabdc34d35f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08407bd5-9ae0-4945-8a2c-1223293ee091", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "88575407f3ee8154555c91bc9d2082ee893abaf7519debc37117b621753df893", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d8e564f-0c72-411d-8795-0b04d5ec63f6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/detr", "title": "DETR", "tokens": 3494, "retrieve_doc": true, "source": "transformers"}, "hash": "9c66f17adb24b4adf81c47bc48208328afb22545b7a85d9b0eb9811e1cc8b002", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11900, "end_char_idx": 14294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For evaluation, one should first convert the
outputs of the model using one of the postprocessing methods of [`~transformers.DetrImageProcessor`]. These can
be be provided to either `CocoEvaluator` or `PanopticEvaluator`, which allow you to calculate metrics like
mean Average Precision (mAP) and Panoptic Quality (PQ). The latter objects are implemented in the [original repository](https://github.com/facebookresearch/detr). See the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR) for more info regarding evaluation.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DETR.

<PipelineTag pipeline="object-detection"/>

- All example notebooks illustrating fine-tuning [`DetrForObjectDetection`] and [`DetrForSegmentation`] on a custom dataset can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/DETR).
- Scripts for finetuning [`DetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DetrConfig

[[autodoc]] DetrConfig

## DetrImageProcessor

[[autodoc]] DetrImageProcessor
    - preprocess
    - post_process_object_detection
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## DetrFeatureExtractor

[[autodoc]] DetrFeatureExtractor
    - __call__
    - post_process_object_detection
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## DETR specific outputs

[[autodoc]] models.detr.modeling_detr.DetrModelOutput

[[autodoc]] models.detr.modeling_detr.DetrObjectDetectionOutput

[[autodoc]] models.detr.modeling_detr.DetrSegmentationOutput

## DetrModel

[[autodoc]] DetrModel
    - forward

## DetrForObjectDetection

[[autodoc]] DetrForObjectDetection
    - forward

## DetrForSegmentation

[[autodoc]] DetrForSegmentation
    - forward


key: _node_content
string_value:
{"id_": "e3d6cb1a-48ce-4345-a881-9471b9d62f69", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers"}, "hash": "3b1fc93b1f1103c5494c134c9e10f5c2ef842dd66b1f399feac95d087cc4acbb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "769dedf0-ca19-4d35-8020-0804f2bf5616", "node_type": "1", "metadata": {}, "hash": "6852a40e77a730b72069745d1c2dca914293210f9552d58125fa06d09af8b6de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLNet

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=xlnet">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlnet-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/xlnet-base-cased">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The XLNet model was proposed in [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov,
Quoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn
bidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization
order.

The abstract from the paper is the following:

*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves
better performance than pretraining approaches based on autoregressive language modeling. However, relying on
corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a
pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive
pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all
permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into
pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large
margin, including question answering, natural language inference, sentiment analysis, and document ranking.*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).

## Usage tips

- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained
  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.
- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
  `target_mapping` inputs to control the attention span and outputs (see examples in
  *examples/pytorch/text-generation/run_generation.py*)
- XLNet is one of the few models that has no sequence length limit.
- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,…,sequence length.
- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies.


key: _node_content
string_value:
{"id_": "769dedf0-ca19-4d35-8020-0804f2bf5616", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ce2f9fba-be50-433d-af58-688a84d9f5d7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers"}, "hash": "3b1fc93b1f1103c5494c134c9e10f5c2ef842dd66b1f399feac95d087cc4acbb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3d6cb1a-48ce-4345-a881-9471b9d62f69", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlnet", "title": "XLNet", "tokens": 1707, "retrieve_doc": true, "source": "transformers"}, "hash": "22a111c93804259ceffc1764e3861399b9f609af9d8118e9b4e6bec61bd7b6f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1577, "end_char_idx": 5969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into
pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large
margin, including question answering, natural language inference, sentiment analysis, and document ranking.*

This model was contributed by [thomwolf](https://huggingface.co/thomwolf). The original code can be found [here](https://github.com/zihangdai/xlnet/).

## Usage tips

- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
- Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained
  using only a sub-set of the output tokens as target which are selected with the `target_mapping` input.
- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
  `target_mapping` inputs to control the attention span and outputs (see examples in
  *examples/pytorch/text-generation/run_generation.py*)
- XLNet is one of the few models that has no sequence length limit.
- XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,…,sequence length.
- XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## XLNetConfig

[[autodoc]] XLNetConfig

## XLNetTokenizer

[[autodoc]] XLNetTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## XLNetTokenizerFast

[[autodoc]] XLNetTokenizerFast

## XLNet specific outputs

[[autodoc]] models.xlnet.modeling_xlnet.XLNetModelOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput

[[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForMultipleChoiceOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForTokenClassificationOutput

[[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput

<frameworkcontent>
<pt>

## XLNetModel

[[autodoc]] XLNetModel
    - forward

## XLNetLMHeadModel

[[autodoc]] XLNetLMHeadModel
    - forward

## XLNetForSequenceClassification

[[autodoc]] XLNetForSequenceClassification
    - forward

## XLNetForMultipleChoice

[[autodoc]] XLNetForMultipleChoice
    - forward

## XLNetForTokenClassification

[[autodoc]] XLNetForTokenClassification
    - forward

## XLNetForQuestionAnsweringSimple

[[autodoc]] XLNetForQuestionAnsweringSimple
    - forward

## XLNetForQuestionAnswering

[[autodoc]] XLNetForQuestionAnswering
    - forward

</pt>
<tf>

## TFXLNetModel

[[autodoc]] TFXLNetModel
    - call

## TFXLNetLMHeadModel

[[autodoc]] TFXLNetLMHeadModel
    - call

## TFXLNetForSequenceClassification

[[autodoc]] TFXLNetForSequenceClassification
    - call

## TFLNetForMultipleChoice

[[autodoc]] TFXLNetForMultipleChoice
    - call

## TFXLNetForTokenClassification

[[autodoc]] TFXLNetForTokenClassification
    - call

## TFXLNetForQuestionAnsweringSimple

[[autodoc]] TFXLNetForQuestionAnsweringSimple
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "9501de56-f184-47ef-bc3f-726e45a32dcf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "hash": "d7db83f3663ada04d53c9f6c94dbd5fc26f849ba45c8d94d4da96501d5ad7087", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "14f61be5-9701-4ddd-8c26-df7770335bdc", "node_type": "1", "metadata": {}, "hash": "0b564a1142000f05d18fb6d5c78e5f38ee094ba146a5af9cefc2af2d15d4a299", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3707, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ESM

## Overview

This page provides code and pre-trained weights for Transformer protein language models from Meta AI's Fundamental 
AI Research Team, providing the state-of-the-art ESMFold and ESM-2, and the previously released ESM-1b and ESM-1v.
Transformer protein language models were introduced in the paper [Biological structure and function emerge from scaling
unsupervised learning to 250 million protein sequences](https://www.pnas.org/content/118/15/e2016239118) by 
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, 
C. Lawrence Zitnick, Jerry Ma, and Rob Fergus.
The first version of this paper was [preprinted in 2019](https://www.biorxiv.org/content/10.1101/622803v1?versioned=true).

ESM-2 outperforms all tested single-sequence protein language models across a range of structure prediction tasks,
and enables atomic resolution structure prediction.
It was released with the paper [Language models of protein sequences at the scale of evolution enable accurate
structure prediction](https://doi.org/10.1101/2022.07.20.500902) by Zeming Lin, Halil Akin, Roshan Rao, Brian Hie,
Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido and Alexander Rives.

Also introduced in this paper was ESMFold. It uses an ESM-2 stem with a head that can predict folded protein
structures with state-of-the-art accuracy. Unlike [AlphaFold2](https://www.nature.com/articles/s41586-021-03819-2),
it relies on the token embeddings from the large pre-trained protein language model stem and does not perform a multiple
sequence alignment (MSA) step at inference time, which means that ESMFold checkpoints are fully "standalone" -
they do not require a database of known protein sequences and structures with associated external query tools
to make predictions, and are much faster as a result.


The abstract from 
"Biological structure and function emerge from scaling unsupervised learning to 250 
million protein sequences" is


*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised
learning has led to major advances in representation learning and statistical generation. In the life sciences, the
anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling
at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To
this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250
million protein sequences spanning evolutionary diversity. The resulting model contains information about biological
properties in its representations. The representations are learned from sequence data alone. The learned representation
space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to
remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and
can be identified by linear projections. Representation learning produces features that generalize across a range of
applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and
improving state-of-the-art features for long-range contact prediction.*


The abstract from
"Language models of protein sequences at the scale of evolution enable accurate structure prediction" is

*Large language models have recently been shown to develop emergent capabilities with scale, going beyond
simple pattern matching to perform higher level reasoning and generate lifelike images and text.


key: _node_content
string_value:
{"id_": "14f61be5-9701-4ddd-8c26-df7770335bdc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "hash": "d7db83f3663ada04d53c9f6c94dbd5fc26f849ba45c8d94d4da96501d5ad7087", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9501de56-f184-47ef-bc3f-726e45a32dcf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "hash": "398d40d9af064a7b61e611f880535dccf931ce505183b510118a27c58747a58d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94770c5c-becf-4258-8797-e1ca2b2b126c", "node_type": "1", "metadata": {}, "hash": "fa43d8f6effcc0541c60b7e2482ddef4cc2520021fc3f1ee6d76d3710289f825", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1887, "end_char_idx": 5415, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract from 
"Biological structure and function emerge from scaling unsupervised learning to 250 
million protein sequences" is


*In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised
learning has led to major advances in representation learning and statistical generation. In the life sciences, the
anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling
at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To
this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250
million protein sequences spanning evolutionary diversity. The resulting model contains information about biological
properties in its representations. The representations are learned from sequence data alone. The learned representation
space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to
remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and
can be identified by linear projections. Representation learning produces features that generalize across a range of
applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and
improving state-of-the-art features for long-range contact prediction.*


The abstract from
"Language models of protein sequences at the scale of evolution enable accurate structure prediction" is

*Large language models have recently been shown to develop emergent capabilities with scale, going beyond
simple pattern matching to perform higher level reasoning and generate lifelike images and text. While
language models trained on protein sequences have been studied at a smaller scale, little is known about
what they learn about biology as they are scaled up. In this work we train models up to 15 billion parameters,
the largest language models of proteins to be evaluated to date. We find that as models are scaled they learn
information enabling the prediction of the three-dimensional structure of a protein at the resolution of
individual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly
from the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for
sequences with low perplexity that are well understood by the language model. ESMFold inference is an
order of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic
proteins in practical timescales.*

The original code can be found [here](https://github.com/facebookresearch/esm) and was
was developed by the Fundamental AI Research team at Meta AI.
ESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)
and [Matt](https://huggingface.co/Rocketknight1).

ESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and
[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their
help throughout the process!

## Usage tips

- ESM models are trained with a masked language modeling (MLM) objective.
- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0.


key: _node_content
string_value:
{"id_": "94770c5c-becf-4258-8797-e1ca2b2b126c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0e3b34c-6620-449f-b66a-4ff56ae82c6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "hash": "d7db83f3663ada04d53c9f6c94dbd5fc26f849ba45c8d94d4da96501d5ad7087", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "14f61be5-9701-4ddd-8c26-df7770335bdc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/esm", "title": "ESM", "tokens": 1680, "retrieve_doc": true, "source": "transformers"}, "hash": "7845f5ed1badd34f1e737c19839d94ffff46cb34167372c3adacd3c6962d8ca6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3872, "end_char_idx": 6570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this work we train models up to 15 billion parameters,
the largest language models of proteins to be evaluated to date. We find that as models are scaled they learn
information enabling the prediction of the three-dimensional structure of a protein at the resolution of
individual atoms. We present ESMFold for high accuracy end-to-end atomic level structure prediction directly
from the individual sequence of a protein. ESMFold has similar accuracy to AlphaFold2 and RoseTTAFold for
sequences with low perplexity that are well understood by the language model. ESMFold inference is an
order of magnitude faster than AlphaFold2, enabling exploration of the structural space of metagenomic
proteins in practical timescales.*

The original code can be found [here](https://github.com/facebookresearch/esm) and was
was developed by the Fundamental AI Research team at Meta AI.
ESM-1b, ESM-1v and ESM-2 were contributed to huggingface by [jasonliu](https://huggingface.co/jasonliu)
and [Matt](https://huggingface.co/Rocketknight1).

ESMFold was contributed to huggingface by [Matt](https://huggingface.co/Rocketknight1) and
[Sylvain](https://huggingface.co/sgugger), with a big thank you to Nikita Smetanin, Roshan Rao and Tom Sercu for their
help throughout the process!

## Usage tips

- ESM models are trained with a masked language modeling (MLM) objective.
- The HuggingFace port of ESMFold uses portions of the [openfold](https://github.com/aqlaboratory/openfold) library. The `openfold` library is licensed under the Apache License 2.0.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Masked language modeling task guide](../tasks/masked_language_modeling)

## EsmConfig

[[autodoc]] EsmConfig
    - all

## EsmTokenizer

[[autodoc]] EsmTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

<frameworkcontent>
<pt>

## EsmModel

[[autodoc]] EsmModel
    - forward

## EsmForMaskedLM

[[autodoc]] EsmForMaskedLM
    - forward

## EsmForSequenceClassification

[[autodoc]] EsmForSequenceClassification
    - forward

## EsmForTokenClassification

[[autodoc]] EsmForTokenClassification
    - forward

## EsmForProteinFolding

[[autodoc]] EsmForProteinFolding
    - forward

</pt>
<tf>

## TFEsmModel

[[autodoc]] TFEsmModel
    - call

## TFEsmForMaskedLM

[[autodoc]] TFEsmForMaskedLM
    - call

## TFEsmForSequenceClassification

[[autodoc]] TFEsmForSequenceClassification
    - call

## TFEsmForTokenClassification

[[autodoc]] TFEsmForTokenClassification
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "a266f8a0-e7f2-4f31-a196-62fcbf76e589", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt", "title": "Pyramid Vision Transformer (PVT)", "tokens": 892, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dae61dce-8f6c-4ac0-837e-4bbd3dd737da", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt", "title": "Pyramid Vision Transformer (PVT)", "tokens": 892, "retrieve_doc": true, "source": "transformers"}, "hash": "e8e7e44c59d35c4df76114714eb0e5267561a2f05094fdc22f06ea678c691588", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Pyramid Vision Transformer (PVT)

## Overview

The PVT model was proposed in
[Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://arxiv.org/abs/2102.12122)
by Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao. The PVT is a type of
vision transformer that utilizes a pyramid structure to make it an effective backbone for dense prediction tasks. Specifically
it allows for more fine-grained inputs (4 x 4 pixels per patch) to be used, while simultaneously shrinking the sequence length
of the Transformer as it deepens - reducing the computational cost. Additionally, a spatial-reduction attention (SRA) layer
is used to further reduce the resource consumption when learning high-resolution features.

The abstract from the paper is the following:

*Although convolutional neural networks (CNNs) have achieved great success in computer vision, this work investigates a 
simpler, convolution-free backbone network useful for many dense prediction tasks. Unlike the recently proposed Vision 
Transformer (ViT) that was designed for image classification specifically, we introduce the Pyramid Vision Transformer 
(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several 
merits compared to current state of the arts. Different from ViT that typically yields low resolution outputs and 
incurs high computational and memory costs, PVT not only can be trained on dense partitions of an image to achieve high 
output resolution, which is important for dense prediction, but also uses a progressive shrinking pyramid to reduce the 
computations of large feature maps. PVT inherits the advantages of both CNN and Transformer, making it a unified 
backbone for various vision tasks without convolutions, where it can be used as a direct replacement for CNN backbones. 
We validate PVT through extensive experiments, showing that it boosts the performance of many downstream tasks, including
object detection, instance and semantic segmentation. For example, with a comparable number of parameters, PVT+RetinaNet 
achieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute AP (see Figure 2). We hope 
that PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future research.*

This model was contributed by [Xrenya](https://huggingface.co/Xrenya). The original code can be found [here](https://github.com/whai362/PVT).


- PVTv1 on ImageNet-1K

| **Model variant**  |**Size** |**Acc@1**|**Params (M)**|
|--------------------|:-------:|:-------:|:------------:|
| PVT-Tiny           |    224  |   75.1  |     13.2     |
| PVT-Small          |    224  |   79.8  |     24.5     |
| PVT-Medium         |    224  |   81.2  |     44.2     |
| PVT-Large          |    224  |   81.7  |     61.4     |


## PvtConfig

[[autodoc]] PvtConfig

## PvtImageProcessor

[[autodoc]] PvtImageProcessor
    - preprocess

## PvtForImageClassification

[[autodoc]] PvtForImageClassification
    - forward

## PvtModel

[[autodoc]] PvtModel
    - forward


key: _node_content
string_value:
{"id_": "73248f00-3189-4269-a4b5-3f6cfdaceb15", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2871f2b5-bab4-4b31-a3d7-ee9c037b7688", "node_type": "1", "metadata": {}, "hash": "d4f73b6a6ca8e6f396f61c8ee559a3b59dcc31c8f41adc78ed9c513735185d76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3102, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Reformer

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=reformer">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-reformer-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/reformer-crime-and-punishment">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The Reformer model was proposed in the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451.pdf) by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.

The abstract from the paper is the following:

*Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can
be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of
Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its
complexity from O(L^2) to O(Llog(L)), where L is the length of the sequence. Furthermore, we use reversible residual
layers instead of the standard residuals, which allows storing activations only once in the training process instead of
N times, where N is the number of layers. The resulting model, the Reformer, performs on par with Transformer models
while being much more memory-efficient and much faster on long sequences.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be
found [here](https://github.com/google/trax/tree/master/trax/models/reformer).

## Usage tips

- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).
- Use Axial position encoding (see below for more details). It’s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.
- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It’s a technique to avoid computing the full product query-key in the attention layers.
- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).
- Compute the feedforward operations by chunks and not on the whole batch.

### Axial Positional Encodings

Axial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embeddings vector of size \\(d\\) being the `config.hidden_size` for
every position \\(i, \ldots, n_s\\), with \\(n_s\\) being `config.max_embedding_size`.


key: _node_content
string_value:
{"id_": "2871f2b5-bab4-4b31-a3d7-ee9c037b7688", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "73248f00-3189-4269-a4b5-3f6cfdaceb15", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "0c927ebf1bfd9e66ff8c93d617cc23d5b5ef0432b8ac021cf6b67043e1fecbac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "75bac14b-4da7-4f30-8166-ade60f6924bd", "node_type": "1", "metadata": {}, "hash": "0f69baf10066f574a8b8d309e1fb789ea227ec2930c8111245a30fb1f3c43cc7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1493, "end_char_idx": 3435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Authors' code can be
found [here](https://github.com/google/trax/tree/master/trax/models/reformer).

## Usage tips

- Reformer does **not** work with *torch.nn.DataParallel* due to a bug in PyTorch, see [issue #36035](https://github.com/pytorch/pytorch/issues/36035).
- Use Axial position encoding (see below for more details). It’s a mechanism to avoid having a huge positional encoding matrix (when the sequence length is very big) by factorizing it into smaller matrices.
- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It’s a technique to avoid computing the full product query-key in the attention layers.
- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).
- Compute the feedforward operations by chunks and not on the whole batch.

### Axial Positional Encodings

Axial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embeddings vector of size \\(d\\) being the `config.hidden_size` for
every position \\(i, \ldots, n_s\\), with \\(n_s\\) being `config.max_embedding_size`. This means that having
a sequence length of \\(n_s = 2^{19} \approx 0.5M\\) and a `config.hidden_size` of \\(d = 2^{10} \approx 1000\\)
would result in a position encoding matrix:

$$X_{i,j}, \text{ with } i \in \left[1,\ldots, d\right] \text{ and } j \in \left[1,\ldots, n_s\right]$$

which alone has over 500M parameters to store.


key: _node_content
string_value:
{"id_": "75bac14b-4da7-4f30-8166-ade60f6924bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2871f2b5-bab4-4b31-a3d7-ee9c037b7688", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "9f9085e179402667ff87ba249c075f07ed2d6b879108b32beef1290eaac66a09", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "444c94a2-2e77-4a42-bc12-b30b1e222ae9", "node_type": "1", "metadata": {}, "hash": "37a387a0577b18f4a59a1f963a979ff706be00179cde55bf3c62d3d399429cd7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1972, "end_char_idx": 4393, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Replace traditional attention by LSH (local-sensitive hashing) attention (see below for more details). It’s a technique to avoid computing the full product query-key in the attention layers.
- Avoid storing the intermediate results of each layer by using reversible transformer layers to obtain them during the backward pass (subtracting the residuals from the input of the next layer gives them back) or recomputing them for results inside a given layer (less efficient than storing them but saves memory).
- Compute the feedforward operations by chunks and not on the whole batch.

### Axial Positional Encodings

Axial Positional Encodings were first implemented in Google's [trax library](https://github.com/google/trax/blob/4d99ad4965bab1deba227539758d59f0df0fef48/trax/layers/research/position_encodings.py#L29)
and developed by the authors of this model's paper. In models that are treating very long input sequences, the
conventional position id encodings store an embeddings vector of size \\(d\\) being the `config.hidden_size` for
every position \\(i, \ldots, n_s\\), with \\(n_s\\) being `config.max_embedding_size`. This means that having
a sequence length of \\(n_s = 2^{19} \approx 0.5M\\) and a `config.hidden_size` of \\(d = 2^{10} \approx 1000\\)
would result in a position encoding matrix:

$$X_{i,j}, \text{ with } i \in \left[1,\ldots, d\right] \text{ and } j \in \left[1,\ldots, n_s\right]$$

which alone has over 500M parameters to store. Axial positional encodings factorize \\(X_{i,j}\\) into two matrices:

$$X^{1}_{i,j}, \text{ with } i \in \left[1,\ldots, d^1\right] \text{ and } j \in \left[1,\ldots, n_s^1\right]$$

and

$$X^{2}_{i,j}, \text{ with } i \in \left[1,\ldots, d^2\right] \text{ and } j \in \left[1,\ldots, n_s^2\right]$$

with:

$$d = d^1 + d^2 \text{ and } n_s = n_s^1 \times n_s^2 .$$

Therefore the following holds:

$$X_{i,j} = \begin{cases}
X^{1}_{i, k}, & \text{if }\ i < d^1 \text{ with } k = j \mod n_s^1 \\
X^{2}_{i - d^1, l}, & \text{if } i \ge d^1 \text{ with } l = \lfloor\frac{j}{n_s^1}\rfloor
\end{cases}$$

Intuitively, this means that a position embedding vector \\(x_j \in \mathbb{R}^{d}\\) is now the composition of two
factorized embedding vectors: \\(x^1_{k, l} + x^2_{l, k}\\), where as the `config.max_embedding_size` dimension
\\(j\\) is factorized into \\(k \text{ and } l\\). This design ensures that each position embedding vector
\\(x_j\\) is unique.


key: _node_content
string_value:
{"id_": "444c94a2-2e77-4a42-bc12-b30b1e222ae9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "75bac14b-4da7-4f30-8166-ade60f6924bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "05e1045a241961de7e6ffe3f8086bc7c1304f8f449372e4bdfc9d56722da2aa2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0bc78568-1aaa-41c5-af62-71b8fa6f7816", "node_type": "1", "metadata": {}, "hash": "ab830199aa7d0cc38601c1afdbf9c4b39534517ae9843d8fd974688c695455d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3436, "end_char_idx": 5018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Axial positional encodings factorize \\(X_{i,j}\\) into two matrices:

$$X^{1}_{i,j}, \text{ with } i \in \left[1,\ldots, d^1\right] \text{ and } j \in \left[1,\ldots, n_s^1\right]$$

and

$$X^{2}_{i,j}, \text{ with } i \in \left[1,\ldots, d^2\right] \text{ and } j \in \left[1,\ldots, n_s^2\right]$$

with:

$$d = d^1 + d^2 \text{ and } n_s = n_s^1 \times n_s^2 .$$

Therefore the following holds:

$$X_{i,j} = \begin{cases}
X^{1}_{i, k}, & \text{if }\ i < d^1 \text{ with } k = j \mod n_s^1 \\
X^{2}_{i - d^1, l}, & \text{if } i \ge d^1 \text{ with } l = \lfloor\frac{j}{n_s^1}\rfloor
\end{cases}$$

Intuitively, this means that a position embedding vector \\(x_j \in \mathbb{R}^{d}\\) is now the composition of two
factorized embedding vectors: \\(x^1_{k, l} + x^2_{l, k}\\), where as the `config.max_embedding_size` dimension
\\(j\\) is factorized into \\(k \text{ and } l\\). This design ensures that each position embedding vector
\\(x_j\\) is unique.

Using the above example again, axial position encoding with \\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\)
can drastically reduced the number of parameters from 500 000 000 to \\(2^{18} + 2^{19} \approx 780 000\\) parameters, this means 85% less memory usage.

In practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\((d^1, d^2)\\) which sum has to be
equal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\((n_s^1, n_s^2)\\) which
product has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence
length* of the `input_ids`.


key: _node_content
string_value:
{"id_": "0bc78568-1aaa-41c5-af62-71b8fa6f7816", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "444c94a2-2e77-4a42-bc12-b30b1e222ae9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ca73e0beccac501b7284d077b1595caf99b989095d60066c8e448ed48d2e644f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "07efaba1-0270-4882-b630-4d3f315d836f", "node_type": "1", "metadata": {}, "hash": "396889b2d203e7e020062d3dd57b0db89efcffcab854cdbf59464f8e729464b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4317, "end_char_idx": 7468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This design ensures that each position embedding vector
\\(x_j\\) is unique.

Using the above example again, axial position encoding with \\(d^1 = 2^9, d^2 = 2^9, n_s^1 = 2^9, n_s^2 = 2^{10}\\)
can drastically reduced the number of parameters from 500 000 000 to \\(2^{18} + 2^{19} \approx 780 000\\) parameters, this means 85% less memory usage.

In practice, the parameter `config.axial_pos_embds_dim` is set to a tuple \\((d^1, d^2)\\) which sum has to be
equal to `config.hidden_size` and `config.axial_pos_shape` is set to a tuple \\((n_s^1, n_s^2)\\) which
product has to be equal to `config.max_embedding_size`, which during training has to be equal to the *sequence
length* of the `input_ids`.


### LSH Self Attention

In Locality sensitive hashing (LSH) self attention the key and query projection weights are tied. Therefore, the key
query embedding vectors are also tied. LSH self attention uses the locality sensitive hashing mechanism proposed in
[Practical and Optimal LSH for Angular Distance](https://arxiv.org/abs/1509.02897) to assign each of the tied key
query embedding vectors to one of `config.num_buckets` possible buckets. The premise is that the more "similar"
key query embedding vectors (in terms of *cosine similarity*) are to each other, the more likely they are assigned to
the same bucket.

The accuracy of the LSH mechanism can be improved by increasing `config.num_hashes` or directly the argument
`num_hashes` of the forward function so that the output of the LSH self attention better approximates the output
of the "normal" full self attention. The buckets are then sorted and chunked into query key embedding vector chunks
each of length `config.lsh_chunk_length`. For each chunk, the query embedding vectors attend to its key vectors
(which are tied to themselves) and to the key embedding vectors of `config.lsh_num_chunks_before` previous
neighboring chunks and `config.lsh_num_chunks_after` following neighboring chunks.

For more information, see the [original Paper](https://arxiv.org/abs/2001.04451) or this great [blog post](https://www.pragmatic.ml/reformer-deep-dive/).

Note that `config.num_buckets` can also be factorized into a list \\((n_{\text{buckets}}^1,
n_{\text{buckets}}^2)\\). This way instead of assigning the query key embedding vectors to one of \\((1,\ldots,
n_{\text{buckets}})\\) they are assigned to one of \\((1-1,\ldots, n_{\text{buckets}}^1-1, \ldots,
1-n_{\text{buckets}}^2, \ldots, n_{\text{buckets}}^1-n_{\text{buckets}}^2)\\). This is crucial for very long sequences to
save memory.

When training a model from scratch, it is recommended to leave `config.num_buckets=None`, so that depending on the
sequence length a good value for `num_buckets` is calculated on the fly. This value will then automatically be
saved in the config and should be reused for inference.

Using LSH self attention, the memory and time complexity of the query-key matmul operation can be reduced from
\\(\mathcal{O}(n_s \times n_s)\\) to \\(\mathcal{O}(n_s \times \log(n_s))\\), which usually represents the memory
and time bottleneck in a transformer model, with \\(n_s\\) being the sequence length.


key: _node_content
string_value:
{"id_": "07efaba1-0270-4882-b630-4d3f315d836f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e4d32f91-d19c-428a-aaa3-d74edf4ac0c5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "ef72e9e0b2d79de83ddbf77dd39bfc00a9c7c400c2a821564ffbb8974948f80b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0bc78568-1aaa-41c5-af62-71b8fa6f7816", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/reformer", "title": "Reformer", "tokens": 2745, "retrieve_doc": true, "source": "transformers"}, "hash": "2dfbced02b076b3d98a9bd0f3190453540c0815f452c01de69b3334f4d83066b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7471, "end_char_idx": 9746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Local Self Attention

Local self attention is essentially a "normal" self attention layer with key, query and value projections, but is
chunked so that in each chunk of length `config.local_chunk_length` the query embedding vectors only attends to
the key embedding vectors in its chunk and to the key embedding vectors of `config.local_num_chunks_before`
previous neighboring chunks and `config.local_num_chunks_after` following neighboring chunks.

Using Local self attention, the memory and time complexity of the query-key matmul operation can be reduced from
\\(\mathcal{O}(n_s \times n_s)\\) to \\(\mathcal{O}(n_s \times \log(n_s))\\), which usually represents the memory
and time bottleneck in a transformer model, with \\(n_s\\) being the sequence length.


### Training

During training, we must ensure that the sequence length is set to a value that can be divided by the least common
multiple of `config.lsh_chunk_length` and `config.local_chunk_length` and that the parameters of the Axial
Positional Encodings are correctly set as described above. Reformer is very memory efficient so that the model can
easily be trained on sequences as long as 64000 tokens.

For training, the [`ReformerModelWithLMHead`] should be used as follows:

```python
input_ids = tokenizer.encode("This is a sentence from the training data", return_tensors="pt")
loss = model(input_ids, labels=input_ids)[0]
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)

## ReformerConfig

[[autodoc]] ReformerConfig

## ReformerTokenizer

[[autodoc]] ReformerTokenizer
    - save_vocabulary

## ReformerTokenizerFast

[[autodoc]] ReformerTokenizerFast

## ReformerModel

[[autodoc]] ReformerModel
    - forward

## ReformerModelWithLMHead

[[autodoc]] ReformerModelWithLMHead
    - forward

## ReformerForMaskedLM

[[autodoc]] ReformerForMaskedLM
    - forward

## ReformerForSequenceClassification

[[autodoc]] ReformerForSequenceClassification
    - forward

## ReformerForQuestionAnswering

[[autodoc]] ReformerForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "ffbb8ab0-ad32-4915-bdbb-ec6716e7548d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers"}, "hash": "20e681968b6e7457e163f2013069be4e090b079ef8878d04648e0a71866b5658", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49de8193-ddc3-43b3-b140-ceba20200f4e", "node_type": "1", "metadata": {}, "hash": "6fe06c8a56f87437b587d671e99662b382783d5338bc58efe976412b0119f52b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1812, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CamemBERT

## Overview

The CamemBERT model was proposed in [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by
[Louis Martin](https://huggingface.co/louismartin), [Benjamin Muller](https://huggingface.co/benjamin-mlr), [Pedro Javier Ortiz Suárez](https://huggingface.co/pjox), Yoann Dupont, Laurent Romary, Éric Villemonte de la
Clergerie, [Djamé Seddah](https://huggingface.co/Djame), and [Benoît Sagot](https://huggingface.co/sagot). It is based on Facebook's RoBERTa model released in 2019. It is a model
trained on 138GB of French text.

The abstract from the paper is the following:

*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available
models have either been trained on English data or on the concatenation of data in multiple languages. This makes
practical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,
we release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the
performance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,
dependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art
for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and
downstream applications for French NLP.*

This model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).

<Tip>

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well 
as the information relative to the inputs and outputs.


key: _node_content
string_value:
{"id_": "49de8193-ddc3-43b3-b140-ceba20200f4e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "263828bf-a6e4-4968-bbd9-9e705a921925", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers"}, "hash": "20e681968b6e7457e163f2013069be4e090b079ef8878d04648e0a71866b5658", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ffbb8ab0-ad32-4915-bdbb-ec6716e7548d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/camembert", "title": "CamemBERT", "tokens": 1113, "retrieve_doc": true, "source": "transformers"}, "hash": "e561fc47685ceaa4e5dfa0dbf78942f7817e6ed4ee8764782756083506a3d68e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 467, "end_char_idx": 3589, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is based on Facebook's RoBERTa model released in 2019. It is a model
trained on 138GB of French text.

The abstract from the paper is the following:

*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available
models have either been trained on English data or on the concatenation of data in multiple languages. This makes
practical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,
we release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the
performance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,
dependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art
for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and
downstream applications for French NLP.*

This model was contributed by [the ALMAnaCH team (Inria)](https://huggingface.co/almanach). The original code can be found [here](https://camembert-model.fr/).

<Tip>

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well 
as the information relative to the inputs and outputs.

</Tip>

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## CamembertConfig

[[autodoc]] CamembertConfig

## CamembertTokenizer

[[autodoc]] CamembertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## CamembertTokenizerFast

[[autodoc]] CamembertTokenizerFast

<frameworkcontent>
<pt>

## CamembertModel

[[autodoc]] CamembertModel

## CamembertForCausalLM

[[autodoc]] CamembertForCausalLM

## CamembertForMaskedLM

[[autodoc]] CamembertForMaskedLM

## CamembertForSequenceClassification

[[autodoc]] CamembertForSequenceClassification

## CamembertForMultipleChoice

[[autodoc]] CamembertForMultipleChoice

## CamembertForTokenClassification

[[autodoc]] CamembertForTokenClassification

## CamembertForQuestionAnswering

[[autodoc]] CamembertForQuestionAnswering

</pt>
<tf>

## TFCamembertModel

[[autodoc]] TFCamembertModel

## TFCamembertForCasualLM

[[autodoc]] TFCamembertForCausalLM

## TFCamembertForMaskedLM

[[autodoc]] TFCamembertForMaskedLM

## TFCamembertForSequenceClassification

[[autodoc]] TFCamembertForSequenceClassification

## TFCamembertForMultipleChoice

[[autodoc]] TFCamembertForMultipleChoice

## TFCamembertForTokenClassification

[[autodoc]] TFCamembertForTokenClassification

## TFCamembertForQuestionAnswering

[[autodoc]] TFCamembertForQuestionAnswering

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "fd60fa62-c495-4fb7-8fac-3664936f7903", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese", "title": "GPT-NeoX-Japanese", "tokens": 867, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9d114f8-8c6d-4ad8-8cd1-c54cede32e9a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese", "title": "GPT-NeoX-Japanese", "tokens": 867, "retrieve_doc": true, "source": "transformers"}, "hash": "41f26dbfc7d9c1657f059fb4f25ab444205b7ebb3354ae530faf9d9ec2cb20c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPT-NeoX-Japanese

## Overview

We introduce GPT-NeoX-Japanese, which is an autoregressive language model for Japanese, trained on top of [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).
Japanese is a unique language with its large vocabulary and a combination of hiragana, katakana, and kanji writing scripts.
To address this distinct structure of the Japanese language, we use a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). We are very grateful to *tanreinama* for open-sourcing this incredibly helpful tokenizer.
Following the recommendations from Google's research on [PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we have removed bias parameters from transformer blocks, achieving better model performance. Please refer [this article](https://medium.com/ml-abeja/training-a-better-gpt-2-93b157662ae4) in detail.

Development of the model was led by [Shinya Otani](https://github.com/SO0529), [Takayoshi Makabe](https://github.com/spider-man-tm), [Anuj Arora](https://github.com/Anuj040), and [Kyo Hattori](https://github.com/go5paopao) from [ABEJA, Inc.](https://www.abejainc.com/). For more information on this model-building activity, please refer [here (ja)](https://tech-blog.abeja.asia/entry/abeja-gpt-project-202207).

### Usage example

The `generate()` method can be used to generate text using GPT NeoX Japanese model.

```python
>>> from transformers import GPTNeoXJapaneseForCausalLM, GPTNeoXJapaneseTokenizer

>>> model = GPTNeoXJapaneseForCausalLM.from_pretrained("abeja/gpt-neox-japanese-2.7b")
>>> tokenizer = GPTNeoXJapaneseTokenizer.from_pretrained("abeja/gpt-neox-japanese-2.7b")

>>> prompt = "人とAIが協調するためには、"

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]

>>> print(gen_text)
人とAIが協調するためには、AIと人が共存し、AIを正しく理解する必要があります。
```

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## GPTNeoXJapaneseConfig

[[autodoc]] GPTNeoXJapaneseConfig

## GPTNeoXJapaneseTokenizer

[[autodoc]] GPTNeoXJapaneseTokenizer

## GPTNeoXJapaneseModel

[[autodoc]] GPTNeoXJapaneseModel
    - forward

## GPTNeoXJapaneseForCausalLM

[[autodoc]] GPTNeoXJapaneseForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "30a213c4-d5f4-4021-8eb5-26a3bfc34f91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mra", "title": "MRA", "tokens": 618, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "09a25eff-c8e4-454c-a80f-19ad85c1cef7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mra", "title": "MRA", "tokens": 618, "retrieve_doc": true, "source": "transformers"}, "hash": "bedabd28fbb4822f335357a25bc84b9467f71bc8b8e004f189d811121c50087d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2004, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MRA

## Overview

The MRA model was proposed in [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://arxiv.org/abs/2207.10284) by Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh.

The abstract from the paper is the following:

*Transformers have emerged as a preferred model for many tasks in natural language processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at https://github.com/mlpen/mra-attention.*

This model was contributed by [novice03](https://huggingface.co/novice03).
The original code can be found [here](https://github.com/mlpen/mra-attention).

## MraConfig

[[autodoc]] MraConfig

## MraModel

[[autodoc]] MraModel
    - forward

## MraForMaskedLM

[[autodoc]] MraForMaskedLM
    - forward

## MraForSequenceClassification

[[autodoc]] MraForSequenceClassification
    - forward

## MraForMultipleChoice

[[autodoc]] MraForMultipleChoice
    - forward

## MraForTokenClassification

[[autodoc]] MraForTokenClassification
    - forward

## MraForQuestionAnswering

[[autodoc]] MraForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "4495a41c-3204-465a-9484-c15544d4f646", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b42fecbc-2ce4-440b-8808-2243881c7bd9", "node_type": "1", "metadata": {}, "hash": "176a40ea098eb50a28e3baf6d0a48e55146e1f8f1b98c3bf3def86b5d3ff85c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Pop2Piano

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/spaces/sweetcocoa/pop2piano">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The Pop2Piano model was proposed in [Pop2Piano : Pop Audio-based Piano Cover Generation](https://arxiv.org/abs/2211.00895) by Jongho Choi and Kyogu Lee.

Piano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great 
expertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you 
can directly generate a cover from a song's audio waveform. It is the first model to directly generate a piano cover 
from pop audio without melody and chord extraction modules. 

Pop2Piano is an encoder-decoder Transformer model based on [T5](https://arxiv.org/pdf/1910.10683.pdf). The input audio 
is transformed to its waveform and passed to the encoder, which transforms it to a latent representation. The decoder 
uses these latent representations to generate token ids in an autoregressive way. Each token id corresponds to one of four 
different token types: time, velocity, note and 'special'. The token ids are then decoded to their equivalent MIDI file.

The abstract from the paper is the following:

*Piano covers of pop music are enjoyed by many people. However, the
task of automatically generating piano covers of pop music is still
understudied. This is partly due to the lack of synchronized
{Pop, Piano Cover} data pairs, which made it challenging to apply
the latest data-intensive deep learning-based methods. To leverage
the power of the data-driven approach, we make a large amount of
paired and synchronized {Pop, Piano Cover} data using an automated
pipeline. In this paper, we present Pop2Piano, a Transformer network
that generates piano covers given waveforms of pop music. To the best
of our knowledge, this is the first model to generate a piano cover
directly from pop audio without using melody and chord extraction
modules. We show that Pop2Piano, trained with our dataset, is capable
of producing plausible piano covers.*

This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/sweetcocoa/pop2piano).

## Usage tips

* To use Pop2Piano, you will need to install the 🤗 Transformers library, as well as the following third party modules:  
```bash
pip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy
```
Please note that you may need to restart your runtime after installation.
* Pop2Piano is an Encoder-Decoder based model like T5.
* Pop2Piano can be used to generate midi-audio files for a given audio sequence.
* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.
* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.


key: _node_content
string_value:
{"id_": "b42fecbc-2ce4-440b-8808-2243881c7bd9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4495a41c-3204-465a-9484-c15544d4f646", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "ac7403faf52ee7217460c36e340963915fc848e93f3102019d04af1f880aeb4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4c2e4d5-015a-4bf8-b9d8-dc306a806761", "node_type": "1", "metadata": {}, "hash": "fecc9aabea6e4a742ba500b56d7e9862610312e364e7698c0b7d6adf18eca638", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1489, "end_char_idx": 4222, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is partly due to the lack of synchronized
{Pop, Piano Cover} data pairs, which made it challenging to apply
the latest data-intensive deep learning-based methods. To leverage
the power of the data-driven approach, we make a large amount of
paired and synchronized {Pop, Piano Cover} data using an automated
pipeline. In this paper, we present Pop2Piano, a Transformer network
that generates piano covers given waveforms of pop music. To the best
of our knowledge, this is the first model to generate a piano cover
directly from pop audio without using melody and chord extraction
modules. We show that Pop2Piano, trained with our dataset, is capable
of producing plausible piano covers.*

This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).
The original code can be found [here](https://github.com/sweetcocoa/pop2piano).

## Usage tips

* To use Pop2Piano, you will need to install the 🤗 Transformers library, as well as the following third party modules:  
```bash
pip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy
```
Please note that you may need to restart your runtime after installation.
* Pop2Piano is an Encoder-Decoder based model like T5.
* Pop2Piano can be used to generate midi-audio files for a given audio sequence.
* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.
* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.

## Examples

- Example using HuggingFace Dataset:

```python
>>> from datasets import load_dataset
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")
>>> ds = load_dataset("sweetcocoa/pop2piano_ci", split="test")

>>> inputs = processor(
...     audio=ds["audio"][0]["array"], sampling_rate=ds["audio"][0]["sampling_rate"], return_tensors="pt"
... )
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example using your own audio file:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> audio, sr = librosa.load("<your_audio_file_here>", sr=44100)  # feel free to change the sr to a suitable value.


key: _node_content
string_value:
{"id_": "e4c2e4d5-015a-4bf8-b9d8-dc306a806761", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b42fecbc-2ce4-440b-8808-2243881c7bd9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "61e90b5a00a803d8834c5008c0b04c1a0c2562f601c647e2d0b7a37cafa33ebd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d298ddb8-792f-4105-a5c9-cc24e4bd1ae2", "node_type": "1", "metadata": {}, "hash": "98694f1f8b94235607e3f01d18ce2b306e6bd7554ca8122f8467b919d7bdd31b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2633, "end_char_idx": 4983, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* Pop2Piano is an Encoder-Decoder based model like T5.
* Pop2Piano can be used to generate midi-audio files for a given audio sequence.
* Choosing different composers in `Pop2PianoForConditionalGeneration.generate()` can lead to variety of different results.
* Setting the sampling rate to 44.1 kHz when loading the audio file can give good performance.
* Though Pop2Piano was mainly trained on Korean Pop music, it also does pretty well on other Western Pop or Hip Hop songs.

## Examples

- Example using HuggingFace Dataset:

```python
>>> from datasets import load_dataset
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")
>>> ds = load_dataset("sweetcocoa/pop2piano_ci", split="test")

>>> inputs = processor(
...     audio=ds["audio"][0]["array"], sampling_rate=ds["audio"][0]["sampling_rate"], return_tensors="pt"
... )
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example using your own audio file:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> audio, sr = librosa.load("<your_audio_file_here>", sr=44100)  # feel free to change the sr to a suitable value.
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors="pt")
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example of processing multiple audio files in batch:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> # feel free to change the sr to a suitable value.


key: _node_content
string_value:
{"id_": "d298ddb8-792f-4105-a5c9-cc24e4bd1ae2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4c2e4d5-015a-4bf8-b9d8-dc306a806761", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "7bdc11e137df517f063d2165c08dc3b3ec6c40c441ff9cae9eec0f12225fb327", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "81e96e8a-6e27-41f9-b97a-7337ba9d2e53", "node_type": "1", "metadata": {}, "hash": "d111db0729e3d39edcd4585a8418407c6ab6d2c475215c82eb65fe86b5fc4bbe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4223, "end_char_idx": 7426, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=audio, sampling_rate=sr, return_tensors="pt")
>>> model_output = model.generate(input_features=inputs["input_features"], composer="composer1")
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"][0]
>>> tokenizer_output.write("./Outputs/midi_output.mid")
```

- Example of processing multiple audio files in batch:

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoProcessor

>>> # feel free to change the sr to a suitable value.
>>> audio1, sr1 = librosa.load("<your_first_audio_file_here>", sr=44100)  
>>> audio2, sr2 = librosa.load("<your_second_audio_file_here>", sr=44100)
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> processor = Pop2PianoProcessor.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = processor(audio=[audio1, audio2], sampling_rate=[sr1, sr2], return_attention_mask=True, return_tensors="pt")
>>> # Since we now generating in batch(2 audios) we must pass the attention_mask
>>> model_output = model.generate(
...     input_features=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     composer="composer1",
... )
>>> tokenizer_output = processor.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"]

>>> # Since we now have 2 generated MIDI files
>>> tokenizer_output[0].write("./Outputs/midi_output1.mid")
>>> tokenizer_output[1].write("./Outputs/midi_output2.mid")
```


- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer

>>> # feel free to change the sr to a suitable value.
>>> audio1, sr1 = librosa.load("<your_first_audio_file_here>", sr=44100)  
>>> audio2, sr2 = librosa.load("<your_second_audio_file_here>", sr=44100)
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained("sweetcocoa/pop2piano")
>>> tokenizer = Pop2PianoTokenizer.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = feature_extractor(
...     audio=[audio1, audio2], 
...     sampling_rate=[sr1, sr2], 
...     return_attention_mask=True, 
...     return_tensors="pt",
... )
>>> # Since we now generating in batch(2 audios) we must pass the attention_mask
>>> model_output = model.generate(
...     input_features=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     composer="composer1",
... )
>>> tokenizer_output = tokenizer.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"]

>>> # Since we now have 2 generated MIDI files
>>> tokenizer_output[0].write("./Outputs/midi_output1.mid")
>>> tokenizer_output[1].write("./Outputs/midi_output2.mid")
```


key: _node_content
string_value:
{"id_": "81e96e8a-6e27-41f9-b97a-7337ba9d2e53", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ed57a1c-ccd7-4b91-90d2-fac7344d7e10", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "c885b945040dff56af7801a539956bdbb4590f910a09f370396a36c733f507d8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d298ddb8-792f-4105-a5c9-cc24e4bd1ae2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pop2piano", "title": "Pop2Piano", "tokens": 2027, "retrieve_doc": true, "source": "transformers"}, "hash": "9377663cf869f027332ede595313b64663756fd9e224c07d3b5b5d9bc3035261", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5981, "end_char_idx": 7814, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Example of processing multiple audio files in batch (Using `Pop2PianoFeatureExtractor` and `Pop2PianoTokenizer`):

```python
>>> import librosa
>>> from transformers import Pop2PianoForConditionalGeneration, Pop2PianoFeatureExtractor, Pop2PianoTokenizer

>>> # feel free to change the sr to a suitable value.
>>> audio1, sr1 = librosa.load("<your_first_audio_file_here>", sr=44100)  
>>> audio2, sr2 = librosa.load("<your_second_audio_file_here>", sr=44100)
>>> model = Pop2PianoForConditionalGeneration.from_pretrained("sweetcocoa/pop2piano")
>>> feature_extractor = Pop2PianoFeatureExtractor.from_pretrained("sweetcocoa/pop2piano")
>>> tokenizer = Pop2PianoTokenizer.from_pretrained("sweetcocoa/pop2piano")

>>> inputs = feature_extractor(
...     audio=[audio1, audio2], 
...     sampling_rate=[sr1, sr2], 
...     return_attention_mask=True, 
...     return_tensors="pt",
... )
>>> # Since we now generating in batch(2 audios) we must pass the attention_mask
>>> model_output = model.generate(
...     input_features=inputs["input_features"],
...     attention_mask=inputs["attention_mask"],
...     composer="composer1",
... )
>>> tokenizer_output = tokenizer.batch_decode(
...     token_ids=model_output, feature_extractor_output=inputs
... )["pretty_midi_objects"]

>>> # Since we now have 2 generated MIDI files
>>> tokenizer_output[0].write("./Outputs/midi_output1.mid")
>>> tokenizer_output[1].write("./Outputs/midi_output2.mid")
```


## Pop2PianoConfig

[[autodoc]] Pop2PianoConfig

## Pop2PianoFeatureExtractor

[[autodoc]] Pop2PianoFeatureExtractor
    - __call__

## Pop2PianoForConditionalGeneration

[[autodoc]] Pop2PianoForConditionalGeneration
    - forward
    - generate

## Pop2PianoTokenizer

[[autodoc]] Pop2PianoTokenizer
    - __call__

## Pop2PianoProcessor

[[autodoc]] Pop2PianoProcessor
    - __call__


key: _node_content
string_value:
{"id_": "59c91e0f-8a44-4116-9b3a-0f05b24df54e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers"}, "hash": "9486051287f6a75b59f9704b8f4b2fd89d0faef074025d4694e77b3fb9d0759e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "de9e4393-d0cb-4e23-b328-7bbb1a92d79e", "node_type": "1", "metadata": {}, "hash": "66d91ab626ece513a4cf1cee0315161574ada194ef8049084d21a6147b7b6218", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3193, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FalconMamba

## Overview

The FalconMamba model was proposed by TII UAE (Technology Innovation Institute) in their release.

The abstract from the paper is the following:

*We present FalconMamba, a new base large language model based on the novel Mamba architecture. FalconMamba is trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure Mamba-based model, FalconMamba surpasses leading open-weight models based on Transformers, such as Mistral 7B, Llama3 8B, and Falcon2 11B. It is on par with Gemma 7B and outperforms models with different architecture designs, such as RecurrentGemma 9B. Currently, FalconMamba is the best-performing Mamba model in the literature at this scale, surpassing both existing Mamba and hybrid Mamba-Transformer models.
Due to its architecture, FalconMamba is significantly faster at inference and requires substantially less memory for long sequence generation. Despite recent studies suggesting that hybrid Mamba-Transformer models outperform pure architecture designs, we argue and demonstrate that the pure Mamba design can achieve similar, even superior results compared to the hybrid design. We make the weights of our implementation of FalconMamba publicly available under a permissive license.*

Tips:

- FalconMamba is mostly based on Mamba architecutre, the same [tips and best practices](./mamba) would be relevant here.

The model has been trained on approximtely 6T tokens consisting a mixture of many data sources such as RefineWeb, Cosmopedia and Math data.

For more details about the training procedure and the architecture, have a look at [the technical paper of FalconMamba]() (coming soon).

# Usage

Below we demonstrate how to use the model:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b")
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b")

input_ids = tokenizer("Hey how are you doing?", return_tensors= "pt")["input_ids"]

out = model.generate(input_ids, max_new_tokens=10)
print(tokenizer.batch_decode(out))
```

The architecture is also compatible with `torch.compile` for faster generation:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b")
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b", torch_dtype=torch.bfloat16).to(0)
model = torch.compile(model)

input_ids = tokenizer("Hey how are you doing?", return_tensors= "pt")["input_ids"]

out = model.generate(input_ids, max_new_tokens=10)
print(tokenizer.batch_decode(out))
```

If you have access to a GPU that is compatible with `bitsandbytes`, you can also quantize the model in 4-bit precision:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b")
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b", quantization_config=quantization_config)

input_ids = tokenizer("Hey how are you doing?


key: _node_content
string_value:
{"id_": "de9e4393-d0cb-4e23-b328-7bbb1a92d79e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c04a1c8-6597-4009-b945-4a679ae3869d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers"}, "hash": "9486051287f6a75b59f9704b8f4b2fd89d0faef074025d4694e77b3fb9d0759e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59c91e0f-8a44-4116-9b3a-0f05b24df54e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/falcon_mamba", "title": "FalconMamba", "tokens": 1178, "retrieve_doc": true, "source": "transformers"}, "hash": "61664343b410738fbd42954652e79e635db8ee867315a67b846831d2ffcd54c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1669, "end_char_idx": 4243, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Usage

Below we demonstrate how to use the model:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b")
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b")

input_ids = tokenizer("Hey how are you doing?", return_tensors= "pt")["input_ids"]

out = model.generate(input_ids, max_new_tokens=10)
print(tokenizer.batch_decode(out))
```

The architecture is also compatible with `torch.compile` for faster generation:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b")
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b", torch_dtype=torch.bfloat16).to(0)
model = torch.compile(model)

input_ids = tokenizer("Hey how are you doing?", return_tensors= "pt")["input_ids"]

out = model.generate(input_ids, max_new_tokens=10)
print(tokenizer.batch_decode(out))
```

If you have access to a GPU that is compatible with `bitsandbytes`, you can also quantize the model in 4-bit precision:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b")
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b", quantization_config=quantization_config)

input_ids = tokenizer("Hey how are you doing?", return_tensors= "pt")["input_ids"]

out = model.generate(input_ids, max_new_tokens=10)
print(tokenizer.batch_decode(out))
```

You can also play with the instruction fine-tuned model:

```python 
from transformers import FalconMambaForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b-instruct")
model = FalconMambaForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b-instruct")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
input_ids = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True).input_ids

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))
```

## FalconMambaConfig

[[autodoc]] FalconMambaConfig

## FalconMambaModel

[[autodoc]] FalconMambaModel
    - forward

## FalconMambaLMHeadModel

[[autodoc]] FalconMambaForCausalLM
    - forward


