# embedding_metadata

key: _node_content
string_value:
{"id_": "02d74bd7-b3e1-4858-a87b-7e11693db4f9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/olmo", "title": "OLMo", "tokens": 782, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4fd2d65a-3ff9-434e-b576-743f5d1f69b4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/olmo", "title": "OLMo", "tokens": 782, "retrieve_doc": true, "source": "transformers"}, "hash": "c24d38db1fbfbff20ee8bdf6009580154a1ae31e9ce8ddc37dee78cee89a5c34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2473, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OLMo

## Overview

The OLMo model was proposed in [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838) by Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi.

OLMo is a series of **O**pen **L**anguage **Mo**dels designed to enable the science of language models. The OLMo models are trained on the Dolma dataset. We release all code, checkpoints, logs (coming soon), and details involved in training these models.

The abstract from the paper is the following:

*Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, this technical report details the first release of OLMo, a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.*

This model was contributed by [shanearora](https://huggingface.co/shanearora).
The original code can be found [here](https://github.com/allenai/OLMo/tree/main/olmo).


## OlmoConfig

[[autodoc]] OlmoConfig

## OlmoModel

[[autodoc]] OlmoModel
    - forward

## OlmoForCausalLM

[[autodoc]] OlmoForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "e8952a57-85ac-4e3a-9453-74a6ba09cb39", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73f39542-474e-4130-8da6-752f41489ec7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30ce2203-8ad4-43eb-bed1-6bb054171a12", "node_type": "1", "metadata": {}, "hash": "99e106084bac918565a3c12c00bc69a426deb4a3c463552a4a4b211adde43dc8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2887, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLM-RoBERTa

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=xlm-roberta">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-xlm--roberta-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/xlm-roberta-base">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

## Overview

The XLM-RoBERTa model was proposed in [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume
Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. It is based on Facebook's
RoBERTa model released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl
data.

The abstract from the paper is the following:

*This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a
wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred
languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly
outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on
XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on
low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We
also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the
trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource
languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing
per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We
will make XLM-R code, data, and models publicly available.*

This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).

## Usage tips

- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does
  not require `lang` tensors to understand which language is used, and should be able to determine the correct
  language from the input ids.
- Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses masked language modeling on sentences coming from one language.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with XLM-RoBERTa.


key: _node_content
string_value:
{"id_": "30ce2203-8ad4-43eb-bed1-6bb054171a12", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73f39542-474e-4130-8da6-752f41489ec7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8952a57-85ac-4e3a-9453-74a6ba09cb39", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "615be13e47ff6572bf1934b05287ebd8ae6813b65c53ab05aab268101c6505d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b873f7b9-896f-4eba-b197-78782cfeb4b8", "node_type": "1", "metadata": {}, "hash": "05d9756023dcb8700a311a9568141a8b78c722816122eb1d5f8c2bcb55ece4be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1436, "end_char_idx": 4777, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
XLM-R performs particularly well on
low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We
also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the
trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource
languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing
per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We
will make XLM-R code, data, and models publicly available.*

This model was contributed by [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).

## Usage tips

- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does
  not require `lang` tensors to understand which language is used, and should be able to determine the correct
  language from the input ids.
- Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses masked language modeling on sentences coming from one language.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with XLM-RoBERTa. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on how to [finetune XLM RoBERTa for multiclass classification with Habana Gaudi on AWS](https://www.philschmid.de/habana-distributed-training)
- [`XLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification](https://huggingface.co/docs/transformers/tasks/sequence_classification) chapter of the 🤗 Hugging Face Task Guides.
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`XLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).


key: _node_content
string_value:
{"id_": "b873f7b9-896f-4eba-b197-78782cfeb4b8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73f39542-474e-4130-8da6-752f41489ec7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30ce2203-8ad4-43eb-bed1-6bb054171a12", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "2add8e5686903b3ade58647c94ba142b7737fbcc3ea8366ddd57cfba7595b8bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26b7fdfa-9902-4535-affd-52991aec2e6a", "node_type": "1", "metadata": {}, "hash": "3dd21a269c51f98ea7d3ae33c38a42845f150db3e8dd08c84ada06a61bbb894c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3016, "end_char_idx": 6328, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on how to [finetune XLM RoBERTa for multiclass classification with Habana Gaudi on AWS](https://www.philschmid.de/habana-distributed-training)
- [`XLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb).
- [`TFXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
- [`FlaxXLMRobertaForSequenceClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification_flax.ipynb).
- [Text classification](https://huggingface.co/docs/transformers/tasks/sequence_classification) chapter of the 🤗 Hugging Face Task Guides.
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`XLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification.ipynb).
- [`TFXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="text-generation"/>

- [`XLMRobertaForCausalLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) chapter of the 🤗 Hugging Face Task Guides.
- [Causal language modeling task guide](../tasks/language_modeling)

<PipelineTag pipeline="fill-mask"/>

- [`XLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).


key: _node_content
string_value:
{"id_": "26b7fdfa-9902-4535-affd-52991aec2e6a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73f39542-474e-4130-8da6-752f41489ec7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b873f7b9-896f-4eba-b197-78782cfeb4b8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "41fb83122d2b71c7ad018ab8014a80c8c533d6d41edfec5d26be4b5897a4669d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16561167-9143-42ef-aa17-4edbfe02c63e", "node_type": "1", "metadata": {}, "hash": "7ac9b404f8f972cd197f72bcc8325504c1a132e5b041419085aecc7b3a5bca1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4778, "end_char_idx": 7924, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`TFXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/token-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/token_classification-tf.ipynb).
- [`FlaxXLMRobertaForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/token-classification).
- [Token classification](https://huggingface.co/course/chapter7/2?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="text-generation"/>

- [`XLMRobertaForCausalLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) chapter of the 🤗 Hugging Face Task Guides.
- [Causal language modeling task guide](../tasks/language_modeling)

<PipelineTag pipeline="fill-mask"/>

- [`XLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#robertabertdistilbert-and-masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).
- [`TFXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling#run_mlmpy) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).
- [`FlaxXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`XLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).


key: _node_content
string_value:
{"id_": "16561167-9143-42ef-aa17-4edbfe02c63e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73f39542-474e-4130-8da6-752f41489ec7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26b7fdfa-9902-4535-affd-52991aec2e6a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "273b518f4fa0057023fd479fbf1d8f3a94f3b40a39bf18761ff3305b1d1259ed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a46d19be-1525-45f3-9fe3-b12e74f06c1c", "node_type": "1", "metadata": {}, "hash": "abbe03ec8c65de72df62e0564b7606f3650996465bbe73078598c3a41a1ed85f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6626, "end_char_idx": 9091, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxXLMRobertaForMaskedLM`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling#masked-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/masked_language_modeling_flax.ipynb).
- [Masked language modeling](https://huggingface.co/course/chapter7/3?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Masked language modeling](../tasks/masked_language_modeling)

<PipelineTag pipeline="question-answering"/>

- [`XLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb).
- [`TFXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).
- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**

- [`XLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFXLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

🚀 Deploy

- A blog post on how to [Deploy Serverless XLM RoBERTa on AWS Lambda](https://www.philschmid.de/multilingual-serverless-xlm-roberta-with-huggingface).

<Tip> 

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well as the information relative to the inputs and outputs.


key: _node_content
string_value:
{"id_": "a46d19be-1525-45f3-9fe3-b12e74f06c1c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73f39542-474e-4130-8da6-752f41489ec7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "563cb84e4f78488e5e2ea179dbe41331ea64d93ed0ce0e5f1dc3552c66587749", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16561167-9143-42ef-aa17-4edbfe02c63e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta", "title": "XLM-RoBERTa", "tokens": 3116, "retrieve_doc": true, "source": "transformers"}, "hash": "7a594156247f2fa2d9fc9d8e8eca831386d0d3d9f00e3dcce6fa001640c9ba5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7757, "end_char_idx": 11332, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [`FlaxXLMRobertaForQuestionAnswering`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering).
- [Question answering](https://huggingface.co/course/chapter7/7?fw=pt) chapter of the 🤗 Hugging Face Course.
- [Question answering task guide](../tasks/question_answering)

**Multiple choice**

- [`XLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb).
- [`TFXLMRobertaForMultipleChoice`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/multiple-choice) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).
- [Multiple choice task guide](../tasks/multiple_choice)

🚀 Deploy

- A blog post on how to [Deploy Serverless XLM RoBERTa on AWS Lambda](https://www.philschmid.de/multilingual-serverless-xlm-roberta-with-huggingface).

<Tip> 

This implementation is the same as RoBERTa. Refer to the [documentation of RoBERTa](roberta) for usage examples as well as the information relative to the inputs and outputs.
</Tip>

## XLMRobertaConfig

[[autodoc]] XLMRobertaConfig

## XLMRobertaTokenizer

[[autodoc]] XLMRobertaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## XLMRobertaTokenizerFast

[[autodoc]] XLMRobertaTokenizerFast

<frameworkcontent>
<pt>

## XLMRobertaModel

[[autodoc]] XLMRobertaModel
    - forward

## XLMRobertaForCausalLM

[[autodoc]] XLMRobertaForCausalLM
    - forward

## XLMRobertaForMaskedLM

[[autodoc]] XLMRobertaForMaskedLM
    - forward

## XLMRobertaForSequenceClassification

[[autodoc]] XLMRobertaForSequenceClassification
    - forward

## XLMRobertaForMultipleChoice

[[autodoc]] XLMRobertaForMultipleChoice
    - forward

## XLMRobertaForTokenClassification

[[autodoc]] XLMRobertaForTokenClassification
    - forward

## XLMRobertaForQuestionAnswering

[[autodoc]] XLMRobertaForQuestionAnswering
    - forward

</pt>
<tf>

## TFXLMRobertaModel

[[autodoc]] TFXLMRobertaModel
    - call

## TFXLMRobertaForCausalLM

[[autodoc]] TFXLMRobertaForCausalLM
    - call

## TFXLMRobertaForMaskedLM

[[autodoc]] TFXLMRobertaForMaskedLM
    - call

## TFXLMRobertaForSequenceClassification

[[autodoc]] TFXLMRobertaForSequenceClassification
    - call

## TFXLMRobertaForMultipleChoice

[[autodoc]] TFXLMRobertaForMultipleChoice
    - call

## TFXLMRobertaForTokenClassification

[[autodoc]] TFXLMRobertaForTokenClassification
    - call

## TFXLMRobertaForQuestionAnswering

[[autodoc]] TFXLMRobertaForQuestionAnswering
    - call

</tf>
<jax>

## FlaxXLMRobertaModel

[[autodoc]] FlaxXLMRobertaModel
    - __call__

## FlaxXLMRobertaForCausalLM

[[autodoc]] FlaxXLMRobertaForCausalLM
    - __call__

## FlaxXLMRobertaForMaskedLM

[[autodoc]] FlaxXLMRobertaForMaskedLM
    - __call__

## FlaxXLMRobertaForSequenceClassification

[[autodoc]] FlaxXLMRobertaForSequenceClassification
    - __call__

## FlaxXLMRobertaForMultipleChoice

[[autodoc]] FlaxXLMRobertaForMultipleChoice
    - __call__

## FlaxXLMRobertaForTokenClassification

[[autodoc]] FlaxXLMRobertaForTokenClassification
    - __call__

## FlaxXLMRobertaForQuestionAnswering

[[autodoc]] FlaxXLMRobertaForQuestionAnswering
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "fe9d6cbf-1f34-477d-bfe3-b2e938ad95df", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f603bb4b-a837-4896-952f-f805bb6d1410", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "f2d57915678ec31d2e39d54ce890a22db5faf6571ff26d3ab437a77dd1829631", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "02288c93-2f30-4de8-8b23-64e51773c6c9", "node_type": "1", "metadata": {}, "hash": "876660f27fc4b678cb5a0b4b1901bc313f02d3a85b06db95b633449bd9ef95fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2671, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DiT

## Overview

DiT was proposed in [DiT: Self-supervised Pre-training for Document Image Transformer](https://arxiv.org/abs/2203.02378) by Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei.
DiT applies the self-supervised objective of [BEiT](beit) (BERT pre-training of Image Transformers) to 42 million document images, allowing for state-of-the-art results on tasks including:

- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of
  400,000 images belonging to one of 16 classes).
- document layout analysis: the [PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet) dataset (a collection of more
  than 360,000 document images constructed by automatically parsing PubMed XML files).
- table detection: the [ICDAR 2019 cTDaR](https://github.com/cndplab-founder/ICDAR2019_cTDaR) dataset (a collection of
  600 training images and 240 testing images).

The abstract from the paper is the following:

*Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 → 92.69), document layout analysis (91.0 → 94.9) and table detection (94.23 → 96.55). *

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dit_architecture.jpg"
alt="drawing" width="600"/> 

<small> Summary of the approach. Taken from the [original paper](https://arxiv.org/abs/2203.02378). </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/dit).

## Usage tips

One can directly use the weights of DiT with the AutoModel API:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/dit-base")
```

This will load the model pre-trained on masked image modeling. Note that this won't include the language modeling head on top, used to predict visual tokens.


key: _node_content
string_value:
{"id_": "02288c93-2f30-4de8-8b23-64e51773c6c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f603bb4b-a837-4896-952f-f805bb6d1410", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "f2d57915678ec31d2e39d54ce890a22db5faf6571ff26d3ab437a77dd1829631", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe9d6cbf-1f34-477d-bfe3-b2e938ad95df", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "80f383e52accb56508999ad93454a1aa07c9c822dedef86b7c2056380fced41e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c06c1d42-7bb2-4c7e-9ada-3cd563f93bb8", "node_type": "1", "metadata": {}, "hash": "8c6628c7afc940abd54804257dbf62455498a0e0ee6e9b3119b0c8272d6f26ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 936, "end_char_idx": 4283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract from the paper is the following:

*Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose DiT, a self-supervised pre-trained Document Image Transformer model using large-scale unlabeled text images for Document AI tasks, which is essential since no supervised counterparts ever exist due to the lack of human labeled document images. We leverage DiT as the backbone network in a variety of vision-based Document AI tasks, including document image classification, document layout analysis, as well as table detection. Experiment results have illustrated that the self-supervised pre-trained DiT model achieves new state-of-the-art results on these downstream tasks, e.g. document image classification (91.11 → 92.69), document layout analysis (91.0 → 94.9) and table detection (94.23 → 96.55). *

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dit_architecture.jpg"
alt="drawing" width="600"/> 

<small> Summary of the approach. Taken from the [original paper](https://arxiv.org/abs/2203.02378). </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/dit).

## Usage tips

One can directly use the weights of DiT with the AutoModel API:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/dit-base")
```

This will load the model pre-trained on masked image modeling. Note that this won't include the language modeling head on top, used to predict visual tokens.

To include the head, you can load the weights into a `BeitForMaskedImageModeling` model, like so:

```python
from transformers import BeitForMaskedImageModeling

model = BeitForMaskedImageModeling.from_pretrained("microsoft/dit-base")
```

You can also load a fine-tuned model from the [hub](https://huggingface.co/models?other=dit), like so:

```python
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained("microsoft/dit-base-finetuned-rvlcdip")
```

This particular checkpoint was fine-tuned on [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/), an important benchmark for document image classification.
A notebook that illustrates inference for document image classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DiT/Inference_with_DiT_(Document_Image_Transformer)_for_document_image_classification.ipynb).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DiT.

<PipelineTag pipeline="image-classification"/>

- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "c06c1d42-7bb2-4c7e-9ada-3cd563f93bb8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f603bb4b-a837-4896-952f-f805bb6d1410", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "f2d57915678ec31d2e39d54ce890a22db5faf6571ff26d3ab437a77dd1829631", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "02288c93-2f30-4de8-8b23-64e51773c6c9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/dit", "title": "DiT", "tokens": 1220, "retrieve_doc": true, "source": "transformers"}, "hash": "33bc599b9ecf99b5be6fe3ccee1eaef38d0eacb86a4ec42b7c6f1baaaaaedc11", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2577, "end_char_idx": 4447, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that this won't include the language modeling head on top, used to predict visual tokens.

To include the head, you can load the weights into a `BeitForMaskedImageModeling` model, like so:

```python
from transformers import BeitForMaskedImageModeling

model = BeitForMaskedImageModeling.from_pretrained("microsoft/dit-base")
```

You can also load a fine-tuned model from the [hub](https://huggingface.co/models?other=dit), like so:

```python
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained("microsoft/dit-base-finetuned-rvlcdip")
```

This particular checkpoint was fine-tuned on [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/), an important benchmark for document image classification.
A notebook that illustrates inference for document image classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DiT/Inference_with_DiT_(Document_Image_Transformer)_for_document_image_classification.ipynb).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DiT.

<PipelineTag pipeline="image-classification"/>

- [`BeitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<Tip>

  As DiT's architecture is equivalent to that of BEiT, one can refer to [BEiT's documentation page](beit) for all tips, code examples and notebooks.
</Tip>


key: _node_content
string_value:
{"id_": "19ea6651-dbf7-4356-ab90-7b6567cd7c1b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a64dfa94-dcee-48d7-8a29-846761aa8660", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "hash": "82abd746c8212e2ff0ead89b54433298e514e295d5da65f48dee7dd9605abe97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "85bf5129-9098-460c-81cf-78182e8d7186", "node_type": "1", "metadata": {}, "hash": "da9fa83b82e1fb342718f431af64877b257802249ba234567691f7a88565c418", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3356, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TVLT

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The TVLT model was proposed in [TVLT: Textless Vision-Language Transformer](https://arxiv.org/abs/2209.14156)
by Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal (the first three authors contributed equally). The Textless Vision-Language Transformer (TVLT) is a model that uses raw visual and audio inputs for vision-and-language representation learning, without using text-specific modules such as tokenization or automatic speech recognition (ASR). It can perform various audiovisual and vision-language tasks like retrieval, question answering, etc.

The abstract from the paper is the following:

*In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text.*

<p align="center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvlt_architecture.png"
alt="drawing" width="600"/>
</p>

<small> TVLT architecture. Taken from the <a href="[https://arxiv.org/abs/2102.03334](https://arxiv.org/abs/2209.14156)">original paper</a>. </small>

The original code can be found [here](https://github.com/zinengtang/TVLT). This model was contributed by [Zineng Tang](https://huggingface.co/ZinengTang).

## Usage tips

- TVLT is a model that takes both `pixel_values` and `audio_values` as input. One can use [`TvltProcessor`] to prepare data for the model.
  This processor wraps an image processor (for the image/video modality) and an audio feature extractor (for the audio modality) into one.
- TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a `pixel_mask` that indicates which pixels are real/padding and `audio_mask` that indicates which audio values are real/padding.
- The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in [ViTMAE](vitmae). The difference is that the model includes embedding layers for the audio modality.
- The PyTorch version of this model is only available in torch 1.10 and higher.


key: _node_content
string_value:
{"id_": "85bf5129-9098-460c-81cf-78182e8d7186", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a64dfa94-dcee-48d7-8a29-846761aa8660", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "hash": "82abd746c8212e2ff0ead89b54433298e514e295d5da65f48dee7dd9605abe97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "19ea6651-dbf7-4356-ab90-7b6567cd7c1b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/tvlt", "title": "TVLT", "tokens": 1057, "retrieve_doc": true, "source": "transformers"}, "hash": "5d7682962b4a1005e76baa9ae84911e048d7db14bbd7e43fd6da3502194555bb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1898, "end_char_idx": 3818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

<p align="center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvlt_architecture.png"
alt="drawing" width="600"/>
</p>

<small> TVLT architecture. Taken from the <a href="[https://arxiv.org/abs/2102.03334](https://arxiv.org/abs/2209.14156)">original paper</a>. </small>

The original code can be found [here](https://github.com/zinengtang/TVLT). This model was contributed by [Zineng Tang](https://huggingface.co/ZinengTang).

## Usage tips

- TVLT is a model that takes both `pixel_values` and `audio_values` as input. One can use [`TvltProcessor`] to prepare data for the model.
  This processor wraps an image processor (for the image/video modality) and an audio feature extractor (for the audio modality) into one.
- TVLT is trained with images/videos and audios of various sizes: the authors resize and crop the input images/videos to 224 and limit the length of audio spectrogram to 2048. To make batching of videos and audios possible, the authors use a `pixel_mask` that indicates which pixels are real/padding and `audio_mask` that indicates which audio values are real/padding.
- The design of TVLT is very similar to that of a standard Vision Transformer (ViT) and masked autoencoder (MAE) as in [ViTMAE](vitmae). The difference is that the model includes embedding layers for the audio modality.
- The PyTorch version of this model is only available in torch 1.10 and higher.

## TvltConfig

[[autodoc]] TvltConfig

## TvltProcessor

[[autodoc]] TvltProcessor
    - __call__

## TvltImageProcessor

[[autodoc]] TvltImageProcessor
    - preprocess

## TvltFeatureExtractor

[[autodoc]] TvltFeatureExtractor
    - __call__

## TvltModel

[[autodoc]] TvltModel
    - forward

## TvltForPreTraining

[[autodoc]] TvltForPreTraining
    - forward

## TvltForAudioVisualClassification

[[autodoc]] TvltForAudioVisualClassification
    - forward


key: _node_content
string_value:
{"id_": "4e2e8cf4-34d3-4184-9f27-5df39818202b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32f27a7c-3954-4a2d-a174-350ea5734035", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "hash": "7cae33e702575ee5a646b9f33883830e70f6c2e50d4bc45131937ccc0cd24916", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff0fee8c-99b3-4d08-9f20-25449dfb2dd4", "node_type": "1", "metadata": {}, "hash": "8494db7d7bea4a45969be730d77aa597aea130a6b8af4b027529345a48f9b07d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Time Series Transformer

## Overview

The Time Series Transformer model is a vanilla encoder-decoder Transformer for time series forecasting.
This model was contributed by [kashif](https://huggingface.co/kashif).

## Usage tips

- Similar to other models in the library, [`TimeSeriesTransformerModel`] is the raw Transformer without any head on top, and [`TimeSeriesTransformerForPrediction`]
adds a distribution head on top of the former, which can be used for time-series forecasting. Note that this is a so-called probabilistic forecasting model, not a
point forecasting model. This means that the model learns a distribution, from which one can sample. The model doesn't directly output values.
- [`TimeSeriesTransformerForPrediction`] consists of 2 blocks: an encoder, which takes a `context_length` of time series values as input (called `past_values`),
and a decoder, which predicts a `prediction_length` of time series values into the future (called `future_values`). During training, one needs to provide
pairs of (`past_values` and `future_values`) to the model.
- In addition to the raw (`past_values` and `future_values`), one typically provides additional features to the model. These can be the following:
    - `past_time_features`: temporal features which the model will add to `past_values`. These serve as "positional encodings" for the Transformer encoder.
    Examples are "day of the month", "month of the year", etc. as scalar values (and then stacked together as a vector).
    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being "day of the month", 8 being "month of the year").
    - `future_time_features`: temporal features which the model will add to `future_values`. These serve as "positional encodings" for the Transformer decoder.
    Examples are "day of the month", "month of the year", etc. as scalar values (and then stacked together as a vector).
    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being "day of the month", 8 being "month of the year").
    - `static_categorical_features`: categorical features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the store ID or region ID that identifies a given time-series.
    Note that these features need to be known for ALL data points (also those in the future).
    - `static_real_features`: real-valued features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the image representation of the product for which you have the time-series values (like the [ResNet](resnet) embedding of a "shoe" picture,
    if your time-series is about the sales of shoes).
    Note that these features need to be known for ALL data points (also those in the future).
- The model is trained using "teacher-forcing", similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the
`future_values` one position to the right as input to the decoder, prepended by the last value of `past_values`. At each time step, the model needs to predict the
next target.


key: _node_content
string_value:
{"id_": "ff0fee8c-99b3-4d08-9f20-25449dfb2dd4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32f27a7c-3954-4a2d-a174-350ea5734035", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "hash": "7cae33e702575ee5a646b9f33883830e70f6c2e50d4bc45131937ccc0cd24916", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e2e8cf4-34d3-4184-9f27-5df39818202b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/time_series_transformer", "title": "Time Series Transformer", "tokens": 1177, "retrieve_doc": true, "source": "transformers"}, "hash": "363d596799887eef9b3e9dc54b9aac6d6ffda92a26007a1dabef3f2843ea3dd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1688, "end_char_idx": 4552, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- `future_time_features`: temporal features which the model will add to `future_values`. These serve as "positional encodings" for the Transformer decoder.
    Examples are "day of the month", "month of the year", etc. as scalar values (and then stacked together as a vector).
    e.g. if a given time-series value was obtained on the 11th of August, then one could have [11, 8] as time feature vector (11 being "day of the month", 8 being "month of the year").
    - `static_categorical_features`: categorical features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the store ID or region ID that identifies a given time-series.
    Note that these features need to be known for ALL data points (also those in the future).
    - `static_real_features`: real-valued features which are static over time (i.e., have the same value for all `past_values` and `future_values`).
    An example here is the image representation of the product for which you have the time-series values (like the [ResNet](resnet) embedding of a "shoe" picture,
    if your time-series is about the sales of shoes).
    Note that these features need to be known for ALL data points (also those in the future).
- The model is trained using "teacher-forcing", similar to how a Transformer is trained for machine translation. This means that, during training, one shifts the
`future_values` one position to the right as input to the decoder, prepended by the last value of `past_values`. At each time step, the model needs to predict the
next target. So the set-up of training is similar to a GPT model for language, except that there's no notion of `decoder_start_token_id` (we just use the last value
of the context as initial input for the decoder).
- At inference time, we give the final value of the `past_values` as input to the decoder. Next, we can sample from the model to make a prediction at the next time step,
which is then fed to the decoder in order to make the next prediction (also called autoregressive generation).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- Check out the Time Series Transformer blog-post in HuggingFace blog: [Probabilistic Time Series Forecasting with 🤗 Transformers](https://huggingface.co/blog/time-series-transformers)


## TimeSeriesTransformerConfig

[[autodoc]] TimeSeriesTransformerConfig

## TimeSeriesTransformerModel

[[autodoc]] TimeSeriesTransformerModel
    - forward

## TimeSeriesTransformerForPrediction

[[autodoc]] TimeSeriesTransformerForPrediction
    - forward


key: _node_content
string_value:
{"id_": "a5fc4d74-6948-4592-9e60-608c24c9ddd3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3a77259a-6e7a-4f39-b87a-572d15334385", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "hash": "eb48b9f8642f3f4ce17193c44211ba0f035de510b36aef07282c1a34a26eb9fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25cd24bf-807f-4424-9a00-b6beeb318485", "node_type": "1", "metadata": {}, "hash": "be364c94058186bd36c8af94c1aae17905e36bb9ddd49274a1afb06915b5065d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPT Neo

## Overview

The GPTNeo model was released in the [EleutherAI/gpt-neo](https://github.com/EleutherAI/gpt-neo) repository by Sid
Black, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy. It is a GPT2 like causal language model trained on the
[Pile](https://pile.eleuther.ai/) dataset.

The architecture is similar to GPT2 except that GPT Neo uses local attention in every other layer with a window size of
256 tokens.

This model was contributed by [valhalla](https://huggingface.co/valhalla).

## Usage example

The `generate()` method can be used to generate text using GPT Neo model.

```python
>>> from transformers import GPTNeoForCausalLM, GPT2Tokenizer

>>> model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")
>>> tokenizer = GPT2Tokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")

>>> prompt = (
...     "In a shocking finding, scientists discovered a herd of unicorns living in a remote, "
...     "previously unexplored valley, in the Andes Mountains. Even more surprising to the "
...     "researchers was the fact that the unicorns spoke perfect English."
... )

>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids

>>> gen_tokens = model.generate(
...     input_ids,
...     do_sample=True,
...     temperature=0.9,
...     max_length=100,
... )
>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

## Combining GPT-Neo and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature, and make sure your hardware is compatible with Flash-Attention 2. More details are available [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) concerning the installation.

Make sure as well to load your model in half-precision (e.g. `torch.float16`).

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"def hello_world():\n    >>> run_script("hello.py")\n    >>> exit(0)\n<|endoftext|>"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `EleutherAI/gpt-neo-2.7B` checkpoint and the Flash Attention 2 version of the model.


key: _node_content
string_value:
{"id_": "25cd24bf-807f-4424-9a00-b6beeb318485", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3a77259a-6e7a-4f39-b87a-572d15334385", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "hash": "eb48b9f8642f3f4ce17193c44211ba0f035de510b36aef07282c1a34a26eb9fb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5fc4d74-6948-4592-9e60-608c24c9ddd3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neo", "title": "GPT Neo", "tokens": 1260, "retrieve_doc": true, "source": "transformers"}, "hash": "de04593e5ecf99ce44ff22fcc0d5ef6e716355fe275e105200c6cc08e565af49", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1589, "end_char_idx": 4065, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
More details are available [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one#flashattention-2) concerning the installation.

Make sure as well to load your model in half-precision (e.g. `torch.float16`).

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
>>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")

>>> prompt = "def hello_world():"

>>> model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
>>> model.to(device)

>>> generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
>>> tokenizer.batch_decode(generated_ids)[0]
"def hello_world():\n    >>> run_script("hello.py")\n    >>> exit(0)\n<|endoftext|>"
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `EleutherAI/gpt-neo-2.7B` checkpoint and the Flash Attention 2 version of the model.
Note that for GPT-Neo it is not possible to train / run on very long context as the max [position embeddings](https://huggingface.co/EleutherAI/gpt-neo-2.7B/blob/main/config.json#L58 ) is limited to 2048 - but this is applicable to all gpt-neo models and not specific to FA-2

<div style="text-align: center">
<img src="https://user-images.githubusercontent.com/49240599/272241893-b1c66b75-3a48-4265-bc47-688448568b3d.png">
</div>


## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Causal language modeling task guide](../tasks/language_modeling)

## GPTNeoConfig

[[autodoc]] GPTNeoConfig


<frameworkcontent>
<pt>

## GPTNeoModel

[[autodoc]] GPTNeoModel
    - forward

## GPTNeoForCausalLM

[[autodoc]] GPTNeoForCausalLM
    - forward

## GPTNeoForQuestionAnswering

[[autodoc]] GPTNeoForQuestionAnswering
    - forward

## GPTNeoForSequenceClassification

[[autodoc]] GPTNeoForSequenceClassification
    - forward

## GPTNeoForTokenClassification

[[autodoc]] GPTNeoForTokenClassification
    - forward

</pt>
<jax>

## FlaxGPTNeoModel

[[autodoc]] FlaxGPTNeoModel
    - __call__

## FlaxGPTNeoForCausalLM

[[autodoc]] FlaxGPTNeoForCausalLM
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "a5db5905-d825-4a01-b8e2-022add8aa9e1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8138146c-01a5-44da-b72b-c825027b21ad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "hash": "a6bad01a61ff44ec9ae62903ef441c2e7ebc1766107ca17dd47b0f07737189c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43634a03-c0b2-407a-a674-a51e7b0caca0", "node_type": "1", "metadata": {}, "hash": "98c38ade1627c5ba36db2365fbeed9b914d7ddc61f9d73fc2cc204a14afa749b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2064, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Hubert

## Overview

Hubert was proposed in [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447) by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan
Salakhutdinov, Abdelrahman Mohamed.

The abstract from the paper is the following:

*Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are
multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training
phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we
propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an
offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our
approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined
acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised
clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means
teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the
state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h,
10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER
reduction on the more challenging dev-other and test-other evaluation subsets.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).

# Usage tips

- Hubert is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- Hubert model was fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
  using [`Wav2Vec2CTCTokenizer`].


key: _node_content
string_value:
{"id_": "43634a03-c0b2-407a-a674-a51e7b0caca0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8138146c-01a5-44da-b72b-c825027b21ad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "hash": "a6bad01a61ff44ec9ae62903ef441c2e7ebc1766107ca17dd47b0f07737189c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5db5905-d825-4a01-b8e2-022add8aa9e1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/hubert", "title": "Hubert", "tokens": 1248, "retrieve_doc": true, "source": "transformers"}, "hash": "91c088db2258fc27afbdaf83b625efdaf3974e8e15c9a51b819829a9d5cb87a7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2067, "end_char_idx": 4488, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Using Flash Attention 2

Flash Attention 2 is an faster, optimized version of the model.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

Below is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of `facebook/hubert-large-ls960-ft`, the flash-attention-2 and the sdpa (scale-dot-product-attention) version. We show the average speedup obtained on the `librispeech_asr` `clean` validation split: 

```python
>>> from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("facebook/hubert-large-ls960-ft", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
...
```

### Expected speedups

Below is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/hubert-large-ls960-ft` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: 


<div style="text-align: center">
<img src="https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Hubert_speedup.png">
</div>


## Resources

- [Audio classification task guide](../tasks/audio_classification)
- [Automatic speech recognition task guide](../tasks/asr)

## HubertConfig

[[autodoc]] HubertConfig

<frameworkcontent>
<pt>

## HubertModel

[[autodoc]] HubertModel
    - forward

## HubertForCTC

[[autodoc]] HubertForCTC
    - forward

## HubertForSequenceClassification

[[autodoc]] HubertForSequenceClassification
    - forward

</pt>
<tf>

## TFHubertModel

[[autodoc]] TFHubertModel
    - call

## TFHubertForCTC

[[autodoc]] TFHubertForCTC
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e3606846-712f-450a-89a2-807214cdf7d0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2", "title": "Qwen2", "tokens": 795, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "53223769-dd46-415c-a5d9-68332076899d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2", "title": "Qwen2", "tokens": 795, "retrieve_doc": true, "source": "transformers"}, "hash": "561d8754208f82f03566c4e3f33cca0b9d34cc8f2e0dbb509b040941e8a9e133", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2374, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Qwen2

## Overview

Qwen2 is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, Qwen2-72B, Qwen2-Audio, etc.

### Model Details

Qwen2 is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.


## Usage tips

`Qwen2-7B` and `Qwen2-7B-Instruct` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)

In the following, we demonstrate how to use `Qwen2-7B-Instruct` for the inference. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-7B-Instruct", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B-Instruct")

>>> prompt = "Give me a short introduction to large language model."

>>> messages = [{"role": "user", "content": prompt}]

>>> text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

>>> model_inputs = tokenizer([text], return_tensors="pt").to(device)

>>> generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)

>>> generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]

>>> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

## Qwen2Config

[[autodoc]] Qwen2Config

## Qwen2Tokenizer

[[autodoc]] Qwen2Tokenizer
    - save_vocabulary

## Qwen2TokenizerFast

[[autodoc]] Qwen2TokenizerFast

## Qwen2Model

[[autodoc]] Qwen2Model
    - forward

## Qwen2ForCausalLM

[[autodoc]] Qwen2ForCausalLM
    - forward

## Qwen2ForSequenceClassification

[[autodoc]] Qwen2ForSequenceClassification
    - forward

## Qwen2ForTokenClassification

[[autodoc]] Qwen2ForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "2f7aa40e-1a37-4c8d-a2f6-17846288460b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8006b203-2e2d-48c8-af2a-93b3e2fe8aa6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "176d7f5ae3c2372dde1cfff67a480ebc9f911de0003830c44caf5e374da6fbf1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "70af76c8-c6a2-4b72-9b77-8b035fe3d763", "node_type": "1", "metadata": {}, "hash": "df4e7cbd0e87004a0f349e98dbbafbfff98526700d153be245c358a6a08a2f21", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ZoeDepth

## Overview

The ZoeDepth model was proposed in [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://arxiv.org/abs/2302.12288) by Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller. ZoeDepth extends the [DPT](dpt) framework for metric (also called absolute) depth estimation. ZoeDepth is pre-trained on 12 datasets using relative depth and fine-tuned on two domains (NYU and KITTI) using metric depth. A lightweight head is used with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier.

The abstract from the paper is the following:

*This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png"
alt="drawing" width="600"/>

<small> ZoeDepth architecture. Taken from the <a href="https://arxiv.org/abs/2302.12288">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/isl-org/ZoeDepth).

## Usage tips

- ZoeDepth is an absolute (also called metric) depth estimation model, unlike DPT which is a relative depth estimation model. This means that ZoeDepth is able to estimate depth in metric units like meters.


key: _node_content
string_value:
{"id_": "70af76c8-c6a2-4b72-9b77-8b035fe3d763", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8006b203-2e2d-48c8-af2a-93b3e2fe8aa6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "176d7f5ae3c2372dde1cfff67a480ebc9f911de0003830c44caf5e374da6fbf1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f7aa40e-1a37-4c8d-a2f6-17846288460b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "7117d21dbd2254749fde8eee9e82a610124d77e49ebea2a2afedb80c9903335c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f41bd5e8-bff5-4617-ab95-39be958f41ce", "node_type": "1", "metadata": {}, "hash": "8a8a8d0a1620e5cab6b9d0fb8a5e4dbc071d483f8a97b8424f5332e689fe9ada", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1147, "end_char_idx": 4454, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png"
alt="drawing" width="600"/>

<small> ZoeDepth architecture. Taken from the <a href="https://arxiv.org/abs/2302.12288">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/isl-org/ZoeDepth).

## Usage tips

- ZoeDepth is an absolute (also called metric) depth estimation model, unlike DPT which is a relative depth estimation model. This means that ZoeDepth is able to estimate depth in metric units like meters.

The easiest to perform inference with ZoeDepth is by leveraging the [pipeline API](../main_classes/pipelines.md):

```python
from transformers import pipeline
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

pipe = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")
result = pipe(image)
depth = result["depth"]
```

Alternatively, one can also perform inference using the classes:

```python
from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation
import torch
import numpy as np
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti")
model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti")

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)

# visualize the prediction
output = prediction.squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(formatted)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ZoeDepth.


key: _node_content
string_value:
{"id_": "f41bd5e8-bff5-4617-ab95-39be958f41ce", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8006b203-2e2d-48c8-af2a-93b3e2fe8aa6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "176d7f5ae3c2372dde1cfff67a480ebc9f911de0003830c44caf5e374da6fbf1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "70af76c8-c6a2-4b72-9b77-8b035fe3d763", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/zoedepth", "title": "ZoeDepth", "tokens": 1267, "retrieve_doc": true, "source": "transformers"}, "hash": "895f2a8abe348305b907b3ee3568bff408cc2ebec59c0dd2552502fb1ef0a406", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2841, "end_char_idx": 4824, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The easiest to perform inference with ZoeDepth is by leveraging the [pipeline API](../main_classes/pipelines.md):

```python
from transformers import pipeline
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

pipe = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")
result = pipe(image)
depth = result["depth"]
```

Alternatively, one can also perform inference using the classes:

```python
from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation
import torch
import numpy as np
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti")
model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti")

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)

# visualize the prediction
output = prediction.squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(formatted)
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ZoeDepth.

- A demo notebook regarding inference with ZoeDepth models can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ZoeDepth). 🌎

## ZoeDepthConfig

[[autodoc]] ZoeDepthConfig

## ZoeDepthImageProcessor

[[autodoc]] ZoeDepthImageProcessor
    - preprocess

## ZoeDepthForDepthEstimation

[[autodoc]] ZoeDepthForDepthEstimation
    - forward


key: _node_content
string_value:
{"id_": "87460836-8208-48fd-b6ce-eaca94a4e84e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fe3e93ff-e5a2-4b00-972d-602508bebb52", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd04cbee546fc1ffa9670497c6a1c019aca937c65cfdec8c905d64deb20f251", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a67bd135-7698-4d8f-afc4-559cfe36cf34", "node_type": "1", "metadata": {}, "hash": "23cfd99685570d19c96c88ce554adc2b21fc5d0abab896e08d10eb4d6cdd35d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2823, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LayoutLM

<a id='Overview'></a>

## Overview

The LayoutLM model was proposed in the paper [LayoutLM: Pre-training of Text and Layout for Document Image
Understanding](https://arxiv.org/abs/1912.13318) by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and
Ming Zhou. It's a simple but effective pretraining method of text and layout for document image understanding and
information extraction tasks, such as form understanding and receipt understanding. It obtains state-of-the-art results
on several downstream tasks:

- form understanding: the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (a collection of 199 annotated
  forms comprising more than 30,000 words).
- receipt understanding: the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset (a collection of 626 receipts for
  training and 347 receipts for testing).
- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of
  400,000 images belonging to one of 16 classes).

The abstract from the paper is the following:

*Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the
widespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,
while neglecting layout and style information that is vital for document image understanding. In this paper, we propose
the LayoutLM to jointly model interactions between text and layout information across scanned document images, which is
beneficial for a great number of real-world document image understanding tasks such as information extraction from
scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM.
To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for
document-level pretraining. It achieves new state-of-the-art results in several downstream tasks, including form
understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification
(from 93.07 to 94.42).*

## Usage tips

- In addition to *input_ids*, [`~transformers.LayoutLMModel.forward`] also expects the input `bbox`, which are
  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such
  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where
  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the
  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000
  scale.


key: _node_content
string_value:
{"id_": "a67bd135-7698-4d8f-afc4-559cfe36cf34", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fe3e93ff-e5a2-4b00-972d-602508bebb52", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd04cbee546fc1ffa9670497c6a1c019aca937c65cfdec8c905d64deb20f251", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87460836-8208-48fd-b6ce-eaca94a4e84e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "2faf7b55caca22ddbd5fa27a00fa0f15324178c0e972ddfc3721b8f621b182c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56a19d38-2922-47fa-aeb8-8f237bc53a49", "node_type": "1", "metadata": {}, "hash": "a3d8f9c18c0d0ecda730ba4098dd00f999f0471a8239e6b17aed453831835dfc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1156, "end_char_idx": 3859, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Despite the
widespread use of pretraining models for NLP applications, they almost exclusively focus on text-level manipulation,
while neglecting layout and style information that is vital for document image understanding. In this paper, we propose
the LayoutLM to jointly model interactions between text and layout information across scanned document images, which is
beneficial for a great number of real-world document image understanding tasks such as information extraction from
scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM.
To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for
document-level pretraining. It achieves new state-of-the-art results in several downstream tasks, including form
understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification
(from 93.07 to 94.42).*

## Usage tips

- In addition to *input_ids*, [`~transformers.LayoutLMModel.forward`] also expects the input `bbox`, which are
  the bounding boxes (i.e. 2D-positions) of the input tokens. These can be obtained using an external OCR engine such
  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where
  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the
  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000
  scale. To normalize, you can use the following function:

```python
def normalize_bbox(bbox, width, height):
    return [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
```

Here, `width` and `height` correspond to the width and height of the original document in which the token
occurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:

```python
from PIL import Image

# Document can be a png, jpg, etc. PDFs must be converted to images.
image = Image.open(name_of_your_document).convert("RGB")

width, height = image.size
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "56a19d38-2922-47fa-aeb8-8f237bc53a49", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fe3e93ff-e5a2-4b00-972d-602508bebb52", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd04cbee546fc1ffa9670497c6a1c019aca937c65cfdec8c905d64deb20f251", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a67bd135-7698-4d8f-afc4-559cfe36cf34", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlm", "title": "LayoutLM", "tokens": 1747, "retrieve_doc": true, "source": "transformers"}, "hash": "af7d8b562821a5f551a4bde51033a3740fb5e92578c9ca5780c502acc29c2b41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2321, "end_char_idx": 6448, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These can be obtained using an external OCR engine such
  as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1) format, where
  (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the
  position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on a 0-1000
  scale. To normalize, you can use the following function:

```python
def normalize_bbox(bbox, width, height):
    return [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
```

Here, `width` and `height` correspond to the width and height of the original document in which the token
occurs. Those can be obtained using the Python Image Library (PIL) library for example, as follows:

```python
from PIL import Image

# Document can be a png, jpg, etc. PDFs must be converted to images.
image = Image.open(name_of_your_document).convert("RGB")

width, height = image.size
```

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


<PipelineTag pipeline="document-question-answering" />

- A blog post on [fine-tuning
  LayoutLM for document-understanding using Keras & Hugging Face
  Transformers](https://www.philschmid.de/fine-tuning-layoutlm-keras).

- A blog post on how to [fine-tune LayoutLM for document-understanding using only Hugging Face Transformers](https://www.philschmid.de/fine-tuning-layoutlm).

- A notebook on how to [fine-tune LayoutLM on the FUNSD dataset with image embeddings](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Add_image_embeddings_to_LayoutLM.ipynb).

- See also: [Document question answering task guide](../tasks/document_question_answering)

<PipelineTag pipeline="text-classification" />

- A notebook on how to [fine-tune LayoutLM for sequence classification on the RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification" />

- A notebook on how to [ fine-tune LayoutLM for token classification on the FUNSD dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb).
- [Token classification task guide](../tasks/token_classification)

**Other resources**
- [Masked language modeling task guide](../tasks/masked_language_modeling)

🚀 Deploy

- A blog post on how to [Deploy LayoutLM with Hugging Face Inference Endpoints](https://www.philschmid.de/inference-endpoints-layoutlm).

## LayoutLMConfig

[[autodoc]] LayoutLMConfig

## LayoutLMTokenizer

[[autodoc]] LayoutLMTokenizer

## LayoutLMTokenizerFast

[[autodoc]] LayoutLMTokenizerFast

<frameworkcontent>
<pt>

## LayoutLMModel

[[autodoc]] LayoutLMModel

## LayoutLMForMaskedLM

[[autodoc]] LayoutLMForMaskedLM

## LayoutLMForSequenceClassification

[[autodoc]] LayoutLMForSequenceClassification

## LayoutLMForTokenClassification

[[autodoc]] LayoutLMForTokenClassification

## LayoutLMForQuestionAnswering

[[autodoc]] LayoutLMForQuestionAnswering

</pt>
<tf>

## TFLayoutLMModel

[[autodoc]] TFLayoutLMModel

## TFLayoutLMForMaskedLM

[[autodoc]] TFLayoutLMForMaskedLM

## TFLayoutLMForSequenceClassification

[[autodoc]] TFLayoutLMForSequenceClassification

## TFLayoutLMForTokenClassification

[[autodoc]] TFLayoutLMForTokenClassification

## TFLayoutLMForQuestionAnswering

[[autodoc]] TFLayoutLMForQuestionAnswering

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "f2c63258-44da-4f91-ab06-d855065db9e9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/table-transformer", "title": "Table Transformer", "tokens": 839, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c84408bb-4fba-441d-86a2-0a5e84c7bb7f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/table-transformer", "title": "Table Transformer", "tokens": 839, "retrieve_doc": true, "source": "transformers"}, "hash": "fb46a3182597b5fb1e5e9da231e9f3dc48ddd06f9afa479cb8e52486178b0c57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Table Transformer

## Overview

The Table Transformer model was proposed in [PubTables-1M: Towards comprehensive table extraction from unstructured documents](https://arxiv.org/abs/2110.00061) by
Brandon Smock, Rohith Pesala, Robin Abraham. The authors introduce a new dataset, PubTables-1M, to benchmark progress in table extraction from unstructured documents,
as well as table structure recognition and functional analysis. The authors train 2 [DETR](detr) models, one for table detection and one for table structure recognition, dubbed Table Transformers.

The abstract from the paper is the following:

*Recently, significant progress has been made applying machine learning to the problem of table structure inference and extraction from unstructured documents.
However, one of the greatest challenges remains the creation of datasets with complete, unambiguous ground truth at scale. To address this, we develop a new, more
comprehensive dataset for table extraction, called PubTables-1M. PubTables-1M contains nearly one million tables from scientific articles, supports multiple input
modalities, and contains detailed header and location information for table structures, making it useful for a wide variety of modeling approaches. It also addresses a significant
source of ground truth inconsistency observed in prior datasets called oversegmentation, using a novel canonicalization procedure. We demonstrate that these improvements lead to a
significant increase in training performance and a more reliable estimate of model performance at evaluation for table structure recognition. Further, we show that transformer-based
object detection models trained on PubTables-1M produce excellent results for all three tasks of detection, structure recognition, and functional analysis without the need for any
special customization for these tasks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/table_transformer_architecture.jpeg"
alt="drawing" width="600"/>

<small> Table detection and table structure recognition clarified. Taken from the <a href="https://arxiv.org/abs/2110.00061">original paper</a>. </small>

The authors released 2 models, one for [table detection](https://huggingface.co/microsoft/table-transformer-detection) in 
documents, one for [table structure recognition](https://huggingface.co/microsoft/table-transformer-structure-recognition) 
(the task of recognizing the individual rows, columns etc. in a table).

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be
found [here](https://github.com/microsoft/table-transformer).

## Resources

<PipelineTag pipeline="object-detection"/>

- A demo notebook for the Table Transformer can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Table%20Transformer).
- It turns out padding of images is quite important for detection. An interesting Github thread with replies from the authors can be found [here](https://github.com/microsoft/table-transformer/issues/68).

## TableTransformerConfig

[[autodoc]] TableTransformerConfig

## TableTransformerModel

[[autodoc]] TableTransformerModel
    - forward

## TableTransformerForObjectDetection

[[autodoc]] TableTransformerForObjectDetection
    - forward


key: _node_content
string_value:
{"id_": "3acf676b-1745-4342-a87c-ad2ead97a4e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "04c8d0ba-01ff-490f-b601-6f856d5dd1c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "c3586208308d2f8147c299a64b332cb34ba08c8ef2856d537908a5d7975f1fe2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "219de218-e2a7-45c6-917a-d96f21749bc9", "node_type": "1", "metadata": {}, "hash": "65a7dca4b06655e142721d57bda860399cb9caa0f2ec0ed005b35822edc88417", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3229, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LiLT

## Overview

The LiLT model was proposed in [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://arxiv.org/abs/2202.13669) by Jiapeng Wang, Lianwen Jin, Kai Ding.
LiLT allows to combine any pre-trained RoBERTa text encoder with a lightweight Layout Transformer, to enable [LayoutLM](layoutlm)-like document understanding for many
languages.

The abstract from the paper is the following:

*Structured document understanding has attracted considerable attention and made significant progress recently, owing to its crucial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured document understanding. LiLT can be pre-trained on the structured documents of a single language and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight languages have shown that LiLT can achieve competitive or even superior performance on diverse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout structure.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/lilt_architecture.jpg"
alt="drawing" width="600"/>

<small> LiLT architecture. Taken from the <a href="https://arxiv.org/abs/2202.13669">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/jpwang/lilt).

## Usage tips

- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).
The script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):

```python
from transformers import LiltModel

model = LiltModel.from_pretrained("path_to_your_files")
model.push_to_hub("name_of_repo_on_the_hub")
```

- When preparing data for the model, make sure to use the token vocabulary that corresponds to the RoBERTa checkpoint you combined with the Layout Transformer.
- As [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) uses the same vocabulary as [LayoutLMv3](layoutlmv3), one can use [`LayoutLMv3TokenizerFast`] to prepare data for the model.
The same is true for [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-infoxlm-base): one can use [`LayoutXLMTokenizerFast`] for that model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LiLT.

- Demo notebooks for LiLT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LiLT).


key: _node_content
string_value:
{"id_": "219de218-e2a7-45c6-917a-d96f21749bc9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "04c8d0ba-01ff-490f-b601-6f856d5dd1c2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "c3586208308d2f8147c299a64b332cb34ba08c8ef2856d537908a5d7975f1fe2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3acf676b-1745-4342-a87c-ad2ead97a4e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/lilt", "title": "LiLT", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "5085f6e89221b332dc7fa8712a747c36c56d939f72db23189305c2ebe2ff1c34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1783, "end_char_idx": 4032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/jpwang/lilt).

## Usage tips

- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).
The script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):

```python
from transformers import LiltModel

model = LiltModel.from_pretrained("path_to_your_files")
model.push_to_hub("name_of_repo_on_the_hub")
```

- When preparing data for the model, make sure to use the token vocabulary that corresponds to the RoBERTa checkpoint you combined with the Layout Transformer.
- As [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-roberta-en-base) uses the same vocabulary as [LayoutLMv3](layoutlmv3), one can use [`LayoutLMv3TokenizerFast`] to prepare data for the model.
The same is true for [lilt-roberta-en-base](https://huggingface.co/SCUT-DLVCLab/lilt-infoxlm-base): one can use [`LayoutXLMTokenizerFast`] for that model.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LiLT.

- Demo notebooks for LiLT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LiLT).

**Documentation resources**
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## LiltConfig

[[autodoc]] LiltConfig

## LiltModel

[[autodoc]] LiltModel
    - forward

## LiltForSequenceClassification

[[autodoc]] LiltForSequenceClassification
    - forward

## LiltForTokenClassification

[[autodoc]] LiltForTokenClassification
    - forward

## LiltForQuestionAnswering

[[autodoc]] LiltForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "8714b8f3-b694-4482-9ab6-4acd84962ea4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51fc1d00-4ca1-439c-9700-3fc3e0975e9b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b2f7fbd2-7567-464e-a6b9-96d6f05f43fa", "node_type": "1", "metadata": {}, "hash": "f2ed00b16198c0e1c433a3b2aceee0fd3c152faa917a9aeb417e6de4d34427ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3162, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# M2M100

## Overview

The M2M100 model was proposed in [Beyond English-Centric Multilingual Machine Translation](https://arxiv.org/abs/2010.11125) by Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky,
Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy
Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin.

The abstract from the paper is the following:

*Existing work in translation demonstrated the potential of massively multilingual machine translation by training a
single model able to translate between any pair of languages. However, much of this work is English-Centric by training
only on data which was translated from or to English. While this is supported by large sources of training data, it
does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation
model that can translate directly between any pair of 100 languages. We build and open source a training dataset that
covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how
to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters
to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly
translating between non-English directions while performing competitively to the best single systems of WMT. We
open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.*

This model was contributed by [valhalla](https://huggingface.co/valhalla).


## Usage tips and examples

M2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks. As the model is
multilingual it expects the sequences in a certain format: A special language id token is used as prefix in both the
source and target text. The source text format is `[lang_code] X [eos]`, where `lang_code` is source language
id for source text and target language id for target text, with `X` being the source or target text.

The [`M2M100Tokenizer`] depends on `sentencepiece` so be sure to install it before running the
examples. To install `sentencepiece` run `pip install sentencepiece`.

**Supervised Training**

```python
from transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer

model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="en", tgt_lang="fr")

src_text = "Life is like a box of chocolates."
tgt_text = "La vie est comme une boîte de chocolat."

model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")

loss = model(**model_inputs).loss  # forward pass
```

**Generation**

M2M100 uses the `eos_token_id` as the `decoder_start_token_id` for generation with the target language id 
being forced as the first generated token. To force the target language id as the first generated token, pass the 
*forced_bos_token_id* parameter to the *generate* method.


key: _node_content
string_value:
{"id_": "b2f7fbd2-7567-464e-a6b9-96d6f05f43fa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51fc1d00-4ca1-439c-9700-3fc3e0975e9b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8714b8f3-b694-4482-9ab6-4acd84962ea4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "5cf8ef92fd3e09e4dc7e9b2434498fe68d7298b8c884d328831a4267c77eb675", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee59ae87-faeb-4178-b458-20983414f38d", "node_type": "1", "metadata": {}, "hash": "4d1d84674c6d4a8e0a3c44dd72b321d370a79a8de96ad0844cbdd1f9a2422100", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1690, "end_char_idx": 4294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips and examples

M2M100 is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation tasks. As the model is
multilingual it expects the sequences in a certain format: A special language id token is used as prefix in both the
source and target text. The source text format is `[lang_code] X [eos]`, where `lang_code` is source language
id for source text and target language id for target text, with `X` being the source or target text.

The [`M2M100Tokenizer`] depends on `sentencepiece` so be sure to install it before running the
examples. To install `sentencepiece` run `pip install sentencepiece`.

**Supervised Training**

```python
from transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer

model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="en", tgt_lang="fr")

src_text = "Life is like a box of chocolates."
tgt_text = "La vie est comme une boîte de chocolat."

model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")

loss = model(**model_inputs).loss  # forward pass
```

**Generation**

M2M100 uses the `eos_token_id` as the `decoder_start_token_id` for generation with the target language id 
being forced as the first generated token. To force the target language id as the first generated token, pass the 
*forced_bos_token_id* parameter to the *generate* method. The following example shows how to translate between 
Hindi to French and Chinese to English using the *facebook/m2m100_418M* checkpoint.

```python
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
>>> chinese_text = "生活就像一盒巧克力。"

>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

>>> # translate Hindi to French
>>> tokenizer.src_lang = "hi"
>>> encoded_hi = tokenizer(hi_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id("fr"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"La vie est comme une boîte de chocolat."

>>> # translate Chinese to English
>>> tokenizer.src_lang = "zh"
>>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"Life is like a box of chocolate."


key: _node_content
string_value:
{"id_": "ee59ae87-faeb-4178-b458-20983414f38d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51fc1d00-4ca1-439c-9700-3fc3e0975e9b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b2f7fbd2-7567-464e-a6b9-96d6f05f43fa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "4b8ab3fbbbc6eae92c63260fa533889f11ff753210100364612538dbdfb99f33", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff2e65ef-6960-4fa2-9a26-810795914bd5", "node_type": "1", "metadata": {}, "hash": "e06ce49ff041b581891a189a4ed13b464ed7b51df174ef64942a966d09a09b3b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3033, "end_char_idx": 5700, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To force the target language id as the first generated token, pass the 
*forced_bos_token_id* parameter to the *generate* method. The following example shows how to translate between 
Hindi to French and Chinese to English using the *facebook/m2m100_418M* checkpoint.

```python
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
>>> chinese_text = "生活就像一盒巧克力。"

>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

>>> # translate Hindi to French
>>> tokenizer.src_lang = "hi"
>>> encoded_hi = tokenizer(hi_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id("fr"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"La vie est comme une boîte de chocolat."

>>> # translate Chinese to English
>>> tokenizer.src_lang = "zh"
>>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"Life is like a box of chocolate."
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## M2M100Config

[[autodoc]] M2M100Config

## M2M100Tokenizer

[[autodoc]] M2M100Tokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## M2M100Model

[[autodoc]] M2M100Model
    - forward

## M2M100ForConditionalGeneration

[[autodoc]] M2M100ForConditionalGeneration
    - forward

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.


key: _node_content
string_value:
{"id_": "ff2e65ef-6960-4fa2-9a26-810795914bd5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51fc1d00-4ca1-439c-9700-3fc3e0975e9b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "fb49c533d85f268a2e02338ab0ad6164c296a376d411f03adb924233b5c23a74", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee59ae87-faeb-4178-b458-20983414f38d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/m2m_100", "title": "M2M100", "tokens": 1860, "retrieve_doc": true, "source": "transformers"}, "hash": "e3b98cc6792a447e0ecc270b2ca2fc8ce905e0dec6a682b770fcadcb8c33233f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4295, "end_char_idx": 6748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## M2M100Config

[[autodoc]] M2M100Config

## M2M100Tokenizer

[[autodoc]] M2M100Tokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## M2M100Model

[[autodoc]] M2M100Model
    - forward

## M2M100ForConditionalGeneration

[[autodoc]] M2M100ForConditionalGeneration
    - forward

## Using Flash Attention 2

Flash Attention 2 is a faster, optimized version of the attention scores computation which relies on `cuda` kernels.

### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

### Usage

To load a model using Flash Attention 2, we can pass the argument `attn_implementation="flash_attention_2"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). You can use either `torch.float16` or `torch.bfloat16` precision.

```python
>>> import torch
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda").eval()
>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

>>> # translate Hindi to French
>>> hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
>>> tokenizer.src_lang = "hi"
>>> encoded_hi = tokenizer(hi_text, return_tensors="pt").to("cuda")
>>> generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.get_lang_id("fr"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"La vie est comme une boîte de chocolat."
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation and the Flash Attention 2.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/visheratin/documentation-images/resolve/main/nllb-speedup.webp">
</div>


key: _node_content
string_value:
{"id_": "8f5837b1-38a3-42a6-a57a-c3fc4a071a1a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/instructblipvideo", "title": "InstructBlipVideo", "tokens": 901, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a5451a6-de0d-4d50-bf13-ab86f8e05df9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/instructblipvideo", "title": "InstructBlipVideo", "tokens": 901, "retrieve_doc": true, "source": "transformers"}, "hash": "4a68ff5cb055c594da95aa27fba9e7922dac45edac3ab6c5dcca72acab03fac4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# InstructBlipVideo

## Overview

## Overview

The InstructBLIPVideo is an extension of the models proposed in [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500) by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.
InstructBLIPVideo uses the same architecture as [InstructBLIP](instructblip) and works with the same checkpoints as [InstructBLIP](instructblip). The only difference is the ability to process videos.

The abstract from the paper is the following:

*General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/instructblip_architecture.jpg"
alt="drawing" width="600"/>

<small> InstructBLIPVideo architecture. Taken from the <a href="https://arxiv.org/abs/2305.06500">original paper.</a> </small>

This model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).
The original code can be found [here](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).

## Usage tips

- The model was trained by sampling 4 frames per video, so it's recommended to sample 4 frames

## InstructBlipVideoConfig

[[autodoc]] InstructBlipVideoConfig
    - from_vision_qformer_text_configs

## InstructBlipVideoVisionConfig

[[autodoc]] InstructBlipVideoVisionConfig

## InstructBlipVideoQFormerConfig

[[autodoc]] InstructBlipVideoQFormerConfig

## InstructBlipVideoProcessor

[[autodoc]] InstructBlipVideoProcessor

## InstructBlipVideoImageProcessor

[[autodoc]] InstructBlipVideoImageProcessor
    - preprocess

## InstructBlipVideoVisionModel

[[autodoc]] InstructBlipVideoVisionModel
    - forward

## InstructBlipVideoQFormerModel

[[autodoc]] InstructBlipVideoQFormerModel
    - forward

## InstructBlipVideoForConditionalGeneration

[[autodoc]] InstructBlipVideoForConditionalGeneration
    - forward
    - generate


key: _node_content
string_value:
{"id_": "8d28b4a9-873e-4f20-95db-6424cdb4e90f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "473b61af-c19c-413a-bfb8-010f1aaf5796", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "c837497986765448e67ed048d580f630604c9e5339ffb7ab5fdfbc216044e896", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04e46219-4359-443a-b49b-6a7a15fba63c", "node_type": "1", "metadata": {}, "hash": "c6faede5bac43651ab761145da80d9b11ae36c3346ba3b2d62900780f13b5e2c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OWLv2

## Overview

OWLv2 was proposed in [Scaling Open-Vocabulary Object Detection](https://arxiv.org/abs/2306.09683) by Matthias Minderer, Alexey Gritsenko, Neil Houlsby. OWLv2 scales up [OWL-ViT](owlvit) using self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. This results in large gains over the previous state-of-the-art for zero-shot object detection.

The abstract from the paper is the following:

*Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlv2_overview.png"
alt="drawing" width="600"/>

<small> OWLv2 high-level overview. Taken from the <a href="https://arxiv.org/abs/2306.09683">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage example

OWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images.


key: _node_content
string_value:
{"id_": "04e46219-4359-443a-b49b-6a7a15fba63c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "473b61af-c19c-413a-bfb8-010f1aaf5796", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "c837497986765448e67ed048d580f630604c9e5339ffb7ab5fdfbc216044e896", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d28b4a9-873e-4f20-95db-6424cdb4e90f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "142112662c723186e3aecdb6ce079dff5d3723c67a1a6ee758a83d4c0b8a79f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b60b49bf-e29c-4c12-9825-b6146126399a", "node_type": "1", "metadata": {}, "hash": "6260cc23f840e5d00db96a8448f3ff1d99e9f2dbd99311442ed6e84719240bb4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1705, "end_char_idx": 3485, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/owlv2_overview.png"
alt="drawing" width="600"/>

<small> OWLv2 high-level overview. Taken from the <a href="https://arxiv.org/abs/2306.09683">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage example

OWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`Owlv2Processor`] and [`Owlv2ForObjectDetection`].


key: _node_content
string_value:
{"id_": "b60b49bf-e29c-4c12-9825-b6146126399a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "473b61af-c19c-413a-bfb8-010f1aaf5796", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "c837497986765448e67ed048d580f630604c9e5339ffb7ab5fdfbc216044e896", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04e46219-4359-443a-b49b-6a7a15fba63c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "3c4ce09c5a0cbb3f4b76b8bbfa2e9cf44e350f25d159793fd7617bf9c3bbcf1e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ce631849-80c7-48cb-982a-942c6bd7d7a0", "node_type": "1", "metadata": {}, "hash": "8d1b944ae73367fdc90c704e54ef8a3ba35dd2b3ea9c7b66c61ad002c0c24d8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1903, "end_char_idx": 5229, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2306.09683">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).

## Usage example

OWLv2 is, just like its predecessor [OWL-ViT](owlvit), a zero-shot text-conditioned object detection model. OWL-ViT uses [CLIP](clip) as its multi-modal backbone, with a ViT-like Transformer to get visual features and a causal language model to get the text features. To use CLIP for detection, OWL-ViT removes the final token pooling layer of the vision model and attaches a lightweight classification and box head to each transformer output token. Open-vocabulary classification is enabled by replacing the fixed classification layer weights with the class-name embeddings obtained from the text model. The authors first train CLIP from scratch and fine-tune it end-to-end with the classification and box heads on standard detection datasets using a bipartite matching loss. One or multiple text queries per image can be used to perform zero-shot text-conditioned object detection.

[`Owlv2ImageProcessor`] can be used to resize (or rescale) and normalize images for the model and [`CLIPTokenizer`] is used to encode the text. [`Owlv2Processor`] wraps [`Owlv2ImageProcessor`] and [`CLIPTokenizer`] into a single instance to both encode the text and prepare the images. The following example shows how to perform object detection using [`Owlv2Processor`] and [`Owlv2ForObjectDetection`].

```python
>>> import requests
>>> from PIL import Image
>>> import torch

>>> from transformers import Owlv2Processor, Owlv2ForObjectDetection

>>> processor = Owlv2Processor.from_pretrained("google/owlv2-base-patch16-ensemble")
>>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble")

>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> texts = [["a photo of a cat", "a photo of a dog"]]
>>> inputs = processor(text=texts, images=image, return_tensors="pt")
>>> outputs = model(**inputs)

>>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]
>>> target_sizes = torch.Tensor([image.size[::-1]])
>>> # Convert outputs (bounding boxes and class logits) to Pascal VOC Format (xmin, ymin, xmax, ymax)
>>> results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)
>>> i = 0  # Retrieve predictions for the first image for the corresponding text queries
>>> text = texts[i]
>>> boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
>>> for box, score, label in zip(boxes, scores, labels):
...     box = [round(i, 2) for i in box.tolist()]
...     print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}")
Detected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35]
Detected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13]
```

## Resources

- A demo notebook on using OWLv2 for zero- and one-shot (image-guided) object detection can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/OWLv2).


key: _node_content
string_value:
{"id_": "ce631849-80c7-48cb-982a-942c6bd7d7a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "473b61af-c19c-413a-bfb8-010f1aaf5796", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "c837497986765448e67ed048d580f630604c9e5339ffb7ab5fdfbc216044e896", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b60b49bf-e29c-4c12-9825-b6146126399a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/owlv2", "title": "OWLv2", "tokens": 1744, "retrieve_doc": true, "source": "transformers"}, "hash": "805b4f0a076b48cda2922779e0e14e0f42b299b73ff49e7cb9218e6285362675", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5230, "end_char_idx": 6493, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [Zero-shot object detection task guide](../tasks/zero_shot_object_detection)

<Tip>

The architecture of OWLv2 is identical to [OWL-ViT](owlvit), however the object detection head now also includes an objectness classifier, which predicts the (query-agnostic) likelihood that a predicted box contains an object (as opposed to background). The objectness score can be used to rank or filter predictions independently of text queries.
Usage of OWLv2 is identical to [OWL-ViT](owlvit) with a new, updated image processor ([`Owlv2ImageProcessor`]).

</Tip>

## Owlv2Config

[[autodoc]] Owlv2Config
    - from_text_vision_configs

## Owlv2TextConfig

[[autodoc]] Owlv2TextConfig

## Owlv2VisionConfig

[[autodoc]] Owlv2VisionConfig

## Owlv2ImageProcessor

[[autodoc]] Owlv2ImageProcessor
    - preprocess
    - post_process_object_detection
    - post_process_image_guided_detection

## Owlv2Processor

[[autodoc]] Owlv2Processor

## Owlv2Model

[[autodoc]] Owlv2Model
    - forward
    - get_text_features
    - get_image_features

## Owlv2TextModel

[[autodoc]] Owlv2TextModel
    - forward

## Owlv2VisionModel

[[autodoc]] Owlv2VisionModel
    - forward

## Owlv2ForObjectDetection

[[autodoc]] Owlv2ForObjectDetection
    - forward
    - image_guided_detection


key: _node_content
string_value:
{"id_": "54c32191-78fd-455e-9cbd-7282d05ae41c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "89c32603-6888-4dfb-89c2-9d9304cebf8f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "hash": "0d9fa9a8bad581978aa3719f8d979a267da9f124cdf3dd64aadd0021920b7205", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bf1d20c2-db1e-41a8-bc23-88deb19c4e2a", "node_type": "1", "metadata": {}, "hash": "6e7d5d122c7543867a5533232c5b311e149775a95169dd271d73f10dd1b6ab8b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 393, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Funnel Transformer

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=funnel">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-funnel-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/funnel-transformer-small">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>


key: _node_content
string_value:
{"id_": "bf1d20c2-db1e-41a8-bc23-88deb19c4e2a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "89c32603-6888-4dfb-89c2-9d9304cebf8f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "hash": "0d9fa9a8bad581978aa3719f8d979a267da9f124cdf3dd64aadd0021920b7205", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54c32191-78fd-455e-9cbd-7282d05ae41c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "hash": "eb4a8851554f9524921c0ec8ed967a0d5f0be9e02baa2bfe81fd16db31b4c792", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c6338399-5397-4595-95a7-8785e496aa1e", "node_type": "1", "metadata": {}, "hash": "e9f96315b912f044856534ffc6a09d39fe3c7070fbc2a4a5bec6d5d4626a43b9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Funnel Transformer

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=funnel">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-funnel-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/funnel-transformer-small">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>


## Overview

The Funnel Transformer model was proposed in the paper [Funnel-Transformer: Filtering out Sequential Redundancy for
Efficient Language Processing](https://arxiv.org/abs/2006.03236). It is a bidirectional transformer model, like
BERT, but with a pooling operation after each block of layers, a bit like in traditional convolutional neural networks
(CNN) in computer vision.

The abstract from the paper is the following:

*With the success of language pretraining, it is highly desirable to develop more efficient architectures of good
scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the
much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only
require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which
gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More
importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further
improve the model capacity. In addition, to perform token-level predictions as required by common pretraining
objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence
via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on
a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading
comprehension.*

This model was contributed by [sgugger](https://huggingface.co/sgugger). The original code can be found [here](https://github.com/laiguokun/Funnel-Transformer).

## Usage tips

- Since Funnel Transformer uses pooling, the sequence length of the hidden states changes after each block of layers. This way, their length is divided by 2, which speeds up the computation of the next hidden states.
  The base model therefore has a final sequence length that is a quarter of the original one. This model can be used
  directly for tasks that just require a sentence summary (like sequence classification or multiple choice). For other
  tasks, the full model is used; this full model has a decoder that upsamples the final hidden states to the same
  sequence length as the input.
- For tasks such as classification, this is not a problem, but for tasks like masked language modeling or token classification, we need a hidden state with the same sequence length as the original input. In those cases, the final hidden states are upsampled to the input sequence length and go through two additional layers. That's why there are two versions of each checkpoint. The version suffixed with “-base” contains only the three blocks, while the version without that suffix contains the three blocks and the upsampling head with its additional layers.
- The Funnel Transformer checkpoints are all available with a full version and a base version. The first ones should be
  used for [`FunnelModel`], [`FunnelForPreTraining`],
  [`FunnelForMaskedLM`], [`FunnelForTokenClassification`] and
  [`FunnelForQuestionAnswering`]. The second ones should be used for
  [`FunnelBaseModel`], [`FunnelForSequenceClassification`] and
  [`FunnelForMultipleChoice`].

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)


key: _node_content
string_value:
{"id_": "c6338399-5397-4595-95a7-8785e496aa1e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "89c32603-6888-4dfb-89c2-9d9304cebf8f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "hash": "0d9fa9a8bad581978aa3719f8d979a267da9f124cdf3dd64aadd0021920b7205", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf1d20c2-db1e-41a8-bc23-88deb19c4e2a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/funnel", "title": "Funnel Transformer", "tokens": 1572, "retrieve_doc": true, "source": "transformers"}, "hash": "44aed8f6e15571e7531206a3f18db134e59a6dc8ed459845c1b780a0ad21202d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4084, "end_char_idx": 5832, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## FunnelConfig

[[autodoc]] FunnelConfig

## FunnelTokenizer

[[autodoc]] FunnelTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## FunnelTokenizerFast

[[autodoc]] FunnelTokenizerFast

## Funnel specific outputs

[[autodoc]] models.funnel.modeling_funnel.FunnelForPreTrainingOutput

[[autodoc]] models.funnel.modeling_tf_funnel.TFFunnelForPreTrainingOutput

<frameworkcontent>
<pt>

## FunnelBaseModel

[[autodoc]] FunnelBaseModel
    - forward

## FunnelModel

[[autodoc]] FunnelModel
    - forward

## FunnelModelForPreTraining

[[autodoc]] FunnelForPreTraining
    - forward

## FunnelForMaskedLM

[[autodoc]] FunnelForMaskedLM
    - forward

## FunnelForSequenceClassification

[[autodoc]] FunnelForSequenceClassification
    - forward

## FunnelForMultipleChoice

[[autodoc]] FunnelForMultipleChoice
    - forward

## FunnelForTokenClassification

[[autodoc]] FunnelForTokenClassification
    - forward

## FunnelForQuestionAnswering

[[autodoc]] FunnelForQuestionAnswering
    - forward

</pt>
<tf>

## TFFunnelBaseModel

[[autodoc]] TFFunnelBaseModel
    - call

## TFFunnelModel

[[autodoc]] TFFunnelModel
    - call

## TFFunnelModelForPreTraining

[[autodoc]] TFFunnelForPreTraining
    - call

## TFFunnelForMaskedLM

[[autodoc]] TFFunnelForMaskedLM
    - call

## TFFunnelForSequenceClassification

[[autodoc]] TFFunnelForSequenceClassification
    - call

## TFFunnelForMultipleChoice

[[autodoc]] TFFunnelForMultipleChoice
    - call

## TFFunnelForTokenClassification

[[autodoc]] TFFunnelForTokenClassification
    - call

## TFFunnelForQuestionAnswering

[[autodoc]] TFFunnelForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "26b9bd52-f005-4484-b284-7adf3bf147f4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e6a1fab7-c72f-4c4e-aefd-d6148cf357fc", "node_type": "1", "metadata": {}, "hash": "20adad9b1102008813e6aff516481b568d5262f92f50b5a4f9ed224b0041f43b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2668, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Llama2

## Overview

The Llama2 model was proposed in [LLaMA: Open Foundation and Fine-Tuned Chat Models](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/) by Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushka rMishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing EllenTan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom. It is a collection of foundation language models ranging from 7B to 70B parameters, with checkpoints finetuned for chat application!

The abstract from the paper is the following:

*In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.*

Checkout all Llama2 model checkpoints [here](https://huggingface.co/models?search=llama2).
This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ) with contributions from [Lysandre Debut](https://huggingface.co/lysandre). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).

## Usage tips

<Tip warning={true}>

The `Llama2` models were trained using `bfloat16`, but the original inference uses `float16`.


key: _node_content
string_value:
{"id_": "e6a1fab7-c72f-4c4e-aefd-d6148cf357fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26b9bd52-f005-4484-b284-7adf3bf147f4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "c673d02e8761a809e137cc0204661fb9cc6902d338f385aa1438d61e3668f1e8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "945edff1-5eff-437b-a3ed-c34a11b91b67", "node_type": "1", "metadata": {}, "hash": "c5ec7006ecb8e91eff4b7d46e7234753e6015b976ba4f9a08be1700e73df63a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1233, "end_char_idx": 4272, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is a collection of foundation language models ranging from 7B to 70B parameters, with checkpoints finetuned for chat application!

The abstract from the paper is the following:

*In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.*

Checkout all Llama2 model checkpoints [here](https://huggingface.co/models?search=llama2).
This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ) with contributions from [Lysandre Debut](https://huggingface.co/lysandre). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).

## Usage tips

<Tip warning={true}>

The `Llama2` models were trained using `bfloat16`, but the original inference uses `float16`. The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be
used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. 

The `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. 

Training the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.

</Tip>

Tips:

- Weights for the Llama2 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)
- The architecture is very similar to the first Llama, with the addition of Grouped Query Attention (GQA) following this [paper](https://arxiv.org/pdf/2305.13245.pdf)
- Setting `config.pretraining_tp` to a value different than 1 will activate the more accurate but slower computation of the linear layers, which should better match the original logits.
- The original model uses `pad_id = -1` which means that there is no padding token. We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({"pad_token":"<pad>"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`.


key: _node_content
string_value:
{"id_": "945edff1-5eff-437b-a3ed-c34a11b91b67", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e6a1fab7-c72f-4c4e-aefd-d6148cf357fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "57920798e6c38438755fea811a5f182eab4e35c630958a1a4b4cd66de332046a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e28eff7-eadb-40b2-9c73-13853915ca31", "node_type": "1", "metadata": {}, "hash": "8c4dc8371e1ae68ea66892214dbd8c46b33b9dfc40a9d52fa9e1c393cb9305a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2669, "end_char_idx": 5970, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be
used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. 

The `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. 

Training the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.

</Tip>

Tips:

- Weights for the Llama2 models can be obtained by filling out [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)
- The architecture is very similar to the first Llama, with the addition of Grouped Query Attention (GQA) following this [paper](https://arxiv.org/pdf/2305.13245.pdf)
- Setting `config.pretraining_tp` to a value different than 1 will activate the more accurate but slower computation of the linear layers, which should better match the original logits.
- The original model uses `pad_id = -1` which means that there is no padding token. We can't have the same logic, make sure to add a padding token using `tokenizer.add_special_tokens({"pad_token":"<pad>"})` and resize the token embedding accordingly. You should also set the `model.config.pad_token_id`. The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.
- After filling out the form and gaining access to the model checkpoints, you should be able to use the already converted checkpoints. Otherwise, if you are converting your own model, feel free to use the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

- After conversion, the model and tokenizer can be loaded via:

```python
from transformers import LlamaForCausalLM, LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("/output/path")
model = LlamaForCausalLM.from_pretrained("/output/path")
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.

- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string.


key: _node_content
string_value:
{"id_": "2e28eff7-eadb-40b2-9c73-13853915ca31", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "945edff1-5eff-437b-a3ed-c34a11b91b67", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "ecfe9eef8c4ecdd9f31817dd06c080b3682f222d288ad9c628e1e44b4e3fddf1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "41634dd5-9568-4867-8ff0-17b43a0aeb65", "node_type": "1", "metadata": {}, "hash": "9c8fe80139f963b263e96614110165e3b131990ef025b9676a4d0f59b4dc8e6c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4273, "end_char_idx": 7277, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `embed_tokens` layer of the model is initialized with `self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.config.padding_idx)`, which makes sure that encoding the padding token will output zeros, so passing it when initializing is recommended.
- After filling out the form and gaining access to the model checkpoints, you should be able to use the already converted checkpoints. Otherwise, if you are converting your own model, feel free to use the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

- After conversion, the model and tokenizer can be loaded via:

```python
from transformers import LlamaForCausalLM, LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("/output/path")
model = LlamaForCausalLM.from_pretrained("/output/path")
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 75B model, it's thus 145GB of RAM needed.

- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string.

- When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.


## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LLaMA2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- [Llama 2 is here - get it on Hugging Face](https://huggingface.co/blog/llama2), a blog post about Llama 2 and how to use it with 🤗 Transformers and 🤗 PEFT.
- [LLaMA 2 - Every Resource you need](https://www.philschmid.de/llama-2), a compilation of relevant resources to learn about LLaMA 2 and how to get started quickly.

<PipelineTag pipeline="text-generation"/>

- A [notebook](https://colab.research.google.com/drive/1PEQyJO1-f6j0S_XJ8DV50NkpzasXkrzd?usp=sharing) on how to fine-tune Llama 2 in Google Colab using QLoRA and 4-bit precision.


key: _node_content
string_value:
{"id_": "41634dd5-9568-4867-8ff0-17b43a0aeb65", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e28eff7-eadb-40b2-9c73-13853915ca31", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "6c7ca7c06a69ee86f4ded253b6157eff87c17477f37162bf326facaad7ff121a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "236a95ab-c863-4705-926a-b29c5fbe9b52", "node_type": "1", "metadata": {}, "hash": "b45673d818dcab6095c8cf5d18d9ee8bf386c86b1fc8c27ec942c6e498d0fc95", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5897, "end_char_idx": 8335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Banana"), the tokenizer does not prepend the prefix space to the string.

- When using Flash Attention 2 via `attn_implementation="flash_attention_2"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.


## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LLaMA2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- [Llama 2 is here - get it on Hugging Face](https://huggingface.co/blog/llama2), a blog post about Llama 2 and how to use it with 🤗 Transformers and 🤗 PEFT.
- [LLaMA 2 - Every Resource you need](https://www.philschmid.de/llama-2), a compilation of relevant resources to learn about LLaMA 2 and how to get started quickly.

<PipelineTag pipeline="text-generation"/>

- A [notebook](https://colab.research.google.com/drive/1PEQyJO1-f6j0S_XJ8DV50NkpzasXkrzd?usp=sharing) on how to fine-tune Llama 2 in Google Colab using QLoRA and 4-bit precision. 🌎
- A [notebook](https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing) on how to fine-tune the "Llama-v2-7b-guanaco" model with 4-bit QLoRA and generate Q&A datasets from PDFs. 🌎

<PipelineTag pipeline="text-classification"/>

- A [notebook](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) on how to fine-tune the Llama 2 model with QLoRa, TRL, and Korean text classification dataset. 🌎🇰🇷

⚗️ Optimization
- [Fine-tune Llama 2 with DPO](https://huggingface.co/blog/dpo-trl), a guide to using the TRL library's DPO method to fine tune Llama 2 on a specific dataset.
- [Extended Guide: Instruction-tune Llama 2](https://www.philschmid.de/instruction-tune-llama-2), a guide to training Llama 2 to generate instructions from inputs, transforming the model from instruction-following to instruction-giving.
- A [notebook](https://colab.research.google.com/drive/1SYpgFpcmtIUzdE7pxqknrM4ArCASfkFQ?usp=sharing) on how to fine-tune the Llama 2 model on a personal computer using QLoRa and TRL.


key: _node_content
string_value:
{"id_": "236a95ab-c863-4705-926a-b29c5fbe9b52", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "41634dd5-9568-4867-8ff0-17b43a0aeb65", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "85fcd5cf2402c08b303ffc3b0e01e8730fb4080b38e2b5634396706f6dcba701", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "73622769-41f7-4060-a36f-eafae2fa525d", "node_type": "1", "metadata": {}, "hash": "c18d822fb17a7cd8885b892614de55873e6ce1227443d22b527ff2548f95a187", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7278, "end_char_idx": 9606, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🌎
- A [notebook](https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing) on how to fine-tune the "Llama-v2-7b-guanaco" model with 4-bit QLoRA and generate Q&A datasets from PDFs. 🌎

<PipelineTag pipeline="text-classification"/>

- A [notebook](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) on how to fine-tune the Llama 2 model with QLoRa, TRL, and Korean text classification dataset. 🌎🇰🇷

⚗️ Optimization
- [Fine-tune Llama 2 with DPO](https://huggingface.co/blog/dpo-trl), a guide to using the TRL library's DPO method to fine tune Llama 2 on a specific dataset.
- [Extended Guide: Instruction-tune Llama 2](https://www.philschmid.de/instruction-tune-llama-2), a guide to training Llama 2 to generate instructions from inputs, transforming the model from instruction-following to instruction-giving.
- A [notebook](https://colab.research.google.com/drive/1SYpgFpcmtIUzdE7pxqknrM4ArCASfkFQ?usp=sharing) on how to fine-tune the Llama 2 model on a personal computer using QLoRa and TRL. 🌎

⚡️ Inference
- A [notebook](https://colab.research.google.com/drive/1TC56ArKerXUpbgRy5vM3woRsbTEVNq7h?usp=sharing) on how to quantize the Llama 2 model using GPTQ from the AutoGPTQ library. 🌎
- A [notebook](https://colab.research.google.com/drive/1X1z9Q6domMKl2CnEM0QGHNwidLfR4dW2?usp=sharing) on how to run the Llama 2 Chat Model with 4-bit quantization on a local computer or Google Colab. 🌎

🚀 Deploy
- [Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama2-qlora), a complete guide from setup to QLoRA fine-tuning and deployment on Amazon SageMaker.
- [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm), a guide on using Hugging Face's LLM DLC container for secure and scalable deployment.


## LlamaConfig

[[autodoc]] LlamaConfig


## LlamaTokenizer

[[autodoc]] LlamaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## LlamaTokenizerFast

[[autodoc]] LlamaTokenizerFast
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - update_post_processor
    - save_vocabulary

## LlamaModel

[[autodoc]] LlamaModel
    - forward


key: _node_content
string_value:
{"id_": "73622769-41f7-4060-a36f-eafae2fa525d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "568189a8-c5e1-45a5-acba-620ed6b840a7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "236a95ab-c863-4705-926a-b29c5fbe9b52", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers"}, "hash": "a0c5204ad5ba8b1b4a22fbe7cfd354e401328c75a609689400d0fd0e0867a859", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8529, "end_char_idx": 9765, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🌎
- A [notebook](https://colab.research.google.com/drive/1X1z9Q6domMKl2CnEM0QGHNwidLfR4dW2?usp=sharing) on how to run the Llama 2 Chat Model with 4-bit quantization on a local computer or Google Colab. 🌎

🚀 Deploy
- [Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama2-qlora), a complete guide from setup to QLoRA fine-tuning and deployment on Amazon SageMaker.
- [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm), a guide on using Hugging Face's LLM DLC container for secure and scalable deployment.


## LlamaConfig

[[autodoc]] LlamaConfig


## LlamaTokenizer

[[autodoc]] LlamaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## LlamaTokenizerFast

[[autodoc]] LlamaTokenizerFast
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - update_post_processor
    - save_vocabulary

## LlamaModel

[[autodoc]] LlamaModel
    - forward


## LlamaForCausalLM

[[autodoc]] LlamaForCausalLM
    - forward

## LlamaForSequenceClassification

[[autodoc]] LlamaForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "29da3941-5a7a-4784-b40f-4d95a25c4a9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mctct", "title": "M-CTC-T", "tokens": 810, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "163063de-e925-4436-b875-63f368d8c6ae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mctct", "title": "M-CTC-T", "tokens": 810, "retrieve_doc": true, "source": "transformers"}, "hash": "d06cfad9abf448714c75a66c975b0c352fb67020b779f3ab6bf196e7d757ef28", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2593, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# M-CTC-T

<Tip warning={true}>

This model is in maintenance mode only, so we won't accept any new PRs changing its code.

If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.
You can do so by running the following command: `pip install -U transformers==4.30.0`.

</Tip>

## Overview

The M-CTC-T model was proposed in [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/abs/2111.00161) by Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. The model is a 1B-param transformer encoder, with a CTC head over 8065 character labels and a language identification head over 60 language ID labels. It is trained on Common Voice (version 6.1, December 2020 release) and VoxPopuli. After training on Common Voice and VoxPopuli, the model is trained on Common Voice only. The labels are unnormalized character-level transcripts (punctuation and capitalization are not removed). The model takes as input Mel filterbank features from a 16Khz audio signal.

The abstract from the paper is the following:

*Semi-supervised learning through pseudo-labeling has become a staple of state-of-the-art monolingual
speech recognition systems. In this work, we extend pseudo-labeling to massively multilingual speech
recognition with 60 languages. We propose a simple pseudo-labeling recipe that works well even
with low-resource languages: train a supervised multilingual model, fine-tune it with semi-supervised
learning on a target language, generate pseudo-labels for that language, and train a final model using
pseudo-labels for all languages, either from scratch or by fine-tuning. Experiments on the labeled
Common Voice and unlabeled VoxPopuli datasets show that our recipe can yield a model with better
performance for many languages that also transfers well to LibriSpeech.*

This model was contributed by [cwkeam](https://huggingface.co/cwkeam). The original code can be found [here](https://github.com/flashlight/wav2letter/tree/main/recipes/mling_pl).

## Usage tips

The PyTorch version of this model is only available in torch 1.9 and higher.

## Resources

- [Automatic speech recognition task guide](../tasks/asr)

## MCTCTConfig

[[autodoc]] MCTCTConfig

## MCTCTFeatureExtractor

[[autodoc]] MCTCTFeatureExtractor
    - __call__

## MCTCTProcessor

[[autodoc]] MCTCTProcessor
    - __call__
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## MCTCTModel

[[autodoc]] MCTCTModel
    - forward

## MCTCTForCTC

[[autodoc]] MCTCTForCTC
    - forward


key: _node_content
string_value:
{"id_": "87feb615-29b2-4dd3-87a0-b593e94113a3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8e259c86-dffa-43ec-b304-9add4c48adee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3aeb17ea8f7970d90ef023bdec387aeac404219c8eedc74ac9872caa704178", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eae47d45-eb31-4c02-9902-910fe99a195a", "node_type": "1", "metadata": {}, "hash": "def884d2e42ca979f1be2a5d3a151172a9cf63fea68255e68ee098c15f7b3528", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2114, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Blenderbot Small

Note that [`BlenderbotSmallModel`] and
[`BlenderbotSmallForConditionalGeneration`] are only used in combination with the checkpoint
[facebook/blenderbot-90M](https://huggingface.co/facebook/blenderbot-90M). Larger Blenderbot checkpoints should
instead be used with [`BlenderbotModel`] and
[`BlenderbotForConditionalGeneration`]

## Overview

The Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,
Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.

The abstract of the paper is the following:

*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that
scaling neural models in the number of parameters and the size of the data they are trained on gives improved results,
we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of
skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to
their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent
persona. We show that large scale models can learn these skills when given appropriate training data and choice of
generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models
and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn
dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing
failure cases of our models.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The authors' code can be
found [here](https://github.com/facebookresearch/ParlAI).

## Usage tips

Blenderbot Small is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than 
the left.


key: _node_content
string_value:
{"id_": "eae47d45-eb31-4c02-9902-910fe99a195a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8e259c86-dffa-43ec-b304-9add4c48adee", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3aeb17ea8f7970d90ef023bdec387aeac404219c8eedc74ac9872caa704178", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87feb615-29b2-4dd3-87a0-b593e94113a3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers"}, "hash": "10ead889d19a2d2e49f4170d6262633564bbf0be59b9eb0ea1ff4a8bdea56c0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2117, "end_char_idx": 3419, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## BlenderbotSmallConfig

[[autodoc]] BlenderbotSmallConfig

## BlenderbotSmallTokenizer

[[autodoc]] BlenderbotSmallTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## BlenderbotSmallTokenizerFast

[[autodoc]] BlenderbotSmallTokenizerFast

<frameworkcontent>
<pt>

## BlenderbotSmallModel

[[autodoc]] BlenderbotSmallModel
    - forward

## BlenderbotSmallForConditionalGeneration

[[autodoc]] BlenderbotSmallForConditionalGeneration
    - forward

## BlenderbotSmallForCausalLM

[[autodoc]] BlenderbotSmallForCausalLM
    - forward

</pt>
<tf>

## TFBlenderbotSmallModel

[[autodoc]] TFBlenderbotSmallModel
    - call

## TFBlenderbotSmallForConditionalGeneration

[[autodoc]] TFBlenderbotSmallForConditionalGeneration
    - call

</tf>
<jax>

## FlaxBlenderbotSmallModel

[[autodoc]] FlaxBlenderbotSmallModel
    - __call__
    - encode
    - decode

## FlaxBlenderbotForConditionalGeneration

[[autodoc]] FlaxBlenderbotSmallForConditionalGeneration
    - __call__
    - encode
    - decode

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "ff59fc0c-a0fb-4b0e-a2bc-ba1498d5dc3e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vitmatte", "title": "ViTMatte", "tokens": 784, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bef0794d-0109-421f-b321-8bfa1bcaab3a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vitmatte", "title": "ViTMatte", "tokens": 784, "retrieve_doc": true, "source": "transformers"}, "hash": "184feb2a440ce292f25cd540ed7243d714a25687e1072b5a03de061eed60f10e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2804, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ViTMatte

## Overview

The ViTMatte model was proposed in [Boosting Image Matting with Pretrained Plain Vision Transformers](https://arxiv.org/abs/2305.15272) by Jingfeng Yao, Xinggang Wang, Shusheng Yang, Baoyuan Wang.
ViTMatte leverages plain [Vision Transformers](vit) for the task of image matting, which is the process of accurately estimating the foreground object in images and videos.

The abstract from the paper is the following:

*Recently, plain vision Transformers (ViTs) have shown impressive performance on various computer vision tasks, thanks to their strong modeling capacity and large-scale pretraining. However, they have not yet conquered the problem of image matting. We hypothesize that image matting could also be boosted by ViTs and present a new efficient and robust ViT-based matting system, named ViTMatte. Our method utilizes (i) a hybrid attention mechanism combined with a convolution neck to help ViTs achieve an excellent performance-computation trade-off in matting tasks. (ii) Additionally, we introduce the detail capture module, which just consists of simple lightweight convolutions to complement the detailed information required by matting. To the best of our knowledge, ViTMatte is the first work to unleash the potential of ViT on image matting with concise adaptation. It inherits many superior properties from ViT to matting, including various pretraining strategies, concise architecture design, and flexible inference strategies. We evaluate ViTMatte on Composition-1k and Distinctions-646, the most commonly used benchmark for image matting, our method achieves state-of-the-art performance and outperforms prior matting works by a large margin.*

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/hustvl/ViTMatte).

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vitmatte_architecture.png"
alt="drawing" width="600"/>

<small> ViTMatte high-level overview. Taken from the <a href="https://arxiv.org/abs/2305.15272">original paper.</a> </small>

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViTMatte.

- A demo notebook regarding inference with [`VitMatteForImageMatting`], including background replacement, can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViTMatte).

<Tip>

The model expects both the image and trimap (concatenated) as input. Use [`ViTMatteImageProcessor`] for this purpose.
</Tip>

## VitMatteConfig

[[autodoc]] VitMatteConfig

## VitMatteImageProcessor

[[autodoc]] VitMatteImageProcessor
    - preprocess

## VitMatteForImageMatting

[[autodoc]] VitMatteForImageMatting
    - forward


key: _node_content
string_value:
{"id_": "cc0241d3-e7c3-430c-9c2a-160625c089ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b4bc059d-13f7-4126-8fd1-697bb115bc4b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers"}, "hash": "67053795ea1275807081df9509ab2419d31b1e98b4ee64704806c04ebacf4cac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d74ede6-58c0-40db-95d3-a0906c7804fd", "node_type": "1", "metadata": {}, "hash": "d54478e8ce727dbf775c2dc7b72a0970d59cd293bb6020770cc330ad26a7074e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2486, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BLOOM

## Overview

The BLOOM model has been proposed with its various versions through the [BigScience Workshop](https://bigscience.huggingface.co/). BigScience is inspired by other open science initiatives where researchers have pooled their time and resources to collectively achieve a higher impact.
The architecture of BLOOM is essentially similar to GPT3 (auto-regressive model for next token prediction), but has been trained on 46 different languages and 13 programming languages.
Several smaller versions of the models have been trained on the same dataset. BLOOM is available in the following versions:

- [bloom-560m](https://huggingface.co/bigscience/bloom-560m)
- [bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)
- [bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)
- [bloom-3b](https://huggingface.co/bigscience/bloom-3b)
- [bloom-7b1](https://huggingface.co/bigscience/bloom-7b1)
- [bloom](https://huggingface.co/bigscience/bloom) (176B parameters)

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BLOOM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-generation"/>

- [`BloomForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).

See also:
- [Causal language modeling task guide](../tasks/language_modeling)
- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)


⚡️ Inference
- A blog on [Optimization story: Bloom inference](https://huggingface.co/blog/bloom-inference-optimization).
- A blog on [Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate](https://huggingface.co/blog/bloom-inference-pytorch-scripts).

⚙️ Training
- A blog on [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed).

## BloomConfig

[[autodoc]] BloomConfig
    - all

## BloomTokenizerFast

[[autodoc]] BloomTokenizerFast
    - all


key: _node_content
string_value:
{"id_": "7d74ede6-58c0-40db-95d3-a0906c7804fd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b4bc059d-13f7-4126-8fd1-697bb115bc4b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers"}, "hash": "67053795ea1275807081df9509ab2419d31b1e98b4ee64704806c04ebacf4cac", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc0241d3-e7c3-430c-9c2a-160625c089ae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers"}, "hash": "61e91917fe5664c93e7b4479bc0df5e97a850539af73712d0dc64e0ffc61affb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1992, "end_char_idx": 3070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
⚡️ Inference
- A blog on [Optimization story: Bloom inference](https://huggingface.co/blog/bloom-inference-optimization).
- A blog on [Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate](https://huggingface.co/blog/bloom-inference-pytorch-scripts).

⚙️ Training
- A blog on [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed).

## BloomConfig

[[autodoc]] BloomConfig
    - all

## BloomTokenizerFast

[[autodoc]] BloomTokenizerFast
    - all


<frameworkcontent>
<pt>

## BloomModel

[[autodoc]] BloomModel
    - forward

## BloomForCausalLM

[[autodoc]] BloomForCausalLM
    - forward

## BloomForSequenceClassification

[[autodoc]] BloomForSequenceClassification
    - forward

## BloomForTokenClassification

[[autodoc]] BloomForTokenClassification
    - forward

## BloomForQuestionAnswering

[[autodoc]] BloomForQuestionAnswering
    - forward

</pt>
<jax>

## FlaxBloomModel

[[autodoc]] FlaxBloomModel
    - __call__

## FlaxBloomForCausalLM

[[autodoc]] FlaxBloomForCausalLM
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "591a2e54-9dd4-4003-8f6e-7342463bd416", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bc4513-05ea-49d2-b971-a3527603439e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers"}, "hash": "b6632ae6f721f2c6cc01e0a517a1126f5aee04fb83808055f9425a7c4d6bf4bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74b75333-0133-4053-b59a-75ee8bb32b9c", "node_type": "1", "metadata": {}, "hash": "baa69438b0f3770996ef35a1512dfbe14e961adc2c6696d605b33ddd36bad7d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2755, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Speech2Text2

  <Tip warning={true}>

  This model is in maintenance mode only, we don't accept any new PRs changing its code.
  If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
  You can do so by running the following command: `pip install -U transformers==4.40.2`.

  </Tip>

## Overview

The Speech2Text2 model is used together with [Wav2Vec2](wav2vec2) for Speech Translation models proposed in
[Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://arxiv.org/abs/2104.06678) by
Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, Alexis Conneau.

Speech2Text2 is a *decoder-only* transformer model that can be used with any speech *encoder-only*, such as
[Wav2Vec2](wav2vec2) or [HuBERT](hubert) for Speech-to-Text tasks. Please refer to the
[SpeechEncoderDecoder](speech-encoder-decoder) class on how to combine Speech2Text2 with any speech *encoder-only*
model.

This model was contributed by [Patrick von Platen](https://huggingface.co/patrickvonplaten).

The original code can be found [here](https://github.com/pytorch/fairseq/blob/1f7ef9ed1e1061f8c7f88f8b94c7186834398690/fairseq/models/wav2vec/wav2vec2_asr.py#L266).

## Usage tips

- Speech2Text2 achieves state-of-the-art results on the CoVoST Speech Translation dataset. For more information, see
  the [official models](https://huggingface.co/models?other=speech2text2) .
- Speech2Text2 is always used within the [SpeechEncoderDecoder](speech-encoder-decoder) framework.
- Speech2Text2's tokenizer is based on [fastBPE](https://github.com/glample/fastBPE).

## Inference

Speech2Text2's [`SpeechEncoderDecoderModel`] model accepts raw waveform input values from speech and
makes use of [`~generation.GenerationMixin.generate`] to translate the input speech
autoregressively to the target language.

The [`Wav2Vec2FeatureExtractor`] class is responsible for preprocessing the input speech and
[`Speech2Text2Tokenizer`] decodes the generated target tokens to the target string. The
[`Speech2Text2Processor`] wraps [`Wav2Vec2FeatureExtractor`] and
[`Speech2Text2Tokenizer`] into a single instance to both extract the input features and decode the
predicted token ids.

- Step-by-step Speech Translation

```python
>>> import torch
>>> from transformers import Speech2Text2Processor, SpeechEncoderDecoderModel
>>> from datasets import load_dataset
>>> import soundfile as sf

>>> model = SpeechEncoderDecoderModel.from_pretrained("facebook/s2t-wav2vec2-large-en-de")
>>> processor = Speech2Text2Processor.from_pretrained("facebook/s2t-wav2vec2-large-en-de")


>>> def map_to_array(batch):
...     speech, _ = sf.read(batch["file"])
...     batch["speech"] = speech
...     return batch


key: _node_content
string_value:
{"id_": "74b75333-0133-4053-b59a-75ee8bb32b9c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bc4513-05ea-49d2-b971-a3527603439e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers"}, "hash": "b6632ae6f721f2c6cc01e0a517a1126f5aee04fb83808055f9425a7c4d6bf4bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "591a2e54-9dd4-4003-8f6e-7342463bd416", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers"}, "hash": "172f06374d203cd12fa52539533315d88a0058e164e49fe90134fcc26172b736", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2630, "end_char_idx": 4308, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def map_to_array(batch):
...     speech, _ = sf.read(batch["file"])
...     batch["speech"] = speech
...     return batch


>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.map(map_to_array)

>>> inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
>>> generated_ids = model.generate(inputs=inputs["input_values"], attention_mask=inputs["attention_mask"])

>>> transcription = processor.batch_decode(generated_ids)
```

- Speech Translation via Pipelines

  The automatic speech recognition pipeline can also be used to translate speech in just a couple lines of code

```python
>>> from datasets import load_dataset
>>> from transformers import pipeline

>>> librispeech_en = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> asr = pipeline(
...     "automatic-speech-recognition",
...     model="facebook/s2t-wav2vec2-large-en-de",
...     feature_extractor="facebook/s2t-wav2vec2-large-en-de",
... )

>>> translation_de = asr(librispeech_en[0]["file"])
```

See [model hub](https://huggingface.co/models?filter=speech2text2) to look for Speech2Text2 checkpoints.

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## Speech2Text2Config

[[autodoc]] Speech2Text2Config

## Speech2TextTokenizer

[[autodoc]] Speech2Text2Tokenizer
    - batch_decode
    - decode
    - save_vocabulary

## Speech2Text2Processor

[[autodoc]] Speech2Text2Processor
    - __call__
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## Speech2Text2ForCausalLM

[[autodoc]] Speech2Text2ForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "334a62d1-8866-4d45-80ce-f8e4de7015f4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutxlm", "title": "LayoutXLM", "tokens": 832, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dd534629-04c3-4c7b-940b-934294082cd5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutxlm", "title": "LayoutXLM", "tokens": 832, "retrieve_doc": true, "source": "transformers"}, "hash": "a326e71685d22f9abcd2a2790fae4219c29a260e61080542d5a4791e8503639b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2724, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LayoutXLM

## Overview

LayoutXLM was proposed in [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836) by Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha
Zhang, Furu Wei. It's a multilingual extension of the [LayoutLMv2 model](https://arxiv.org/abs/2012.14740) trained
on 53 languages.

The abstract from the paper is the following:

*Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document
understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In
this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to
bridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also
introduce a multilingual form understanding benchmark dataset named XFUN, which includes form understanding samples in
7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled
for each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA
cross-lingual pre-trained models on the XFUN dataset.*

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm).

## Usage tips and examples

One can directly plug in the weights of LayoutXLM into a LayoutLMv2 model, like so:

```python
from transformers import LayoutLMv2Model

model = LayoutLMv2Model.from_pretrained("microsoft/layoutxlm-base")
```

Note that LayoutXLM has its own tokenizer, based on
[`LayoutXLMTokenizer`]/[`LayoutXLMTokenizerFast`]. You can initialize it as
follows:

```python
from transformers import LayoutXLMTokenizer

tokenizer = LayoutXLMTokenizer.from_pretrained("microsoft/layoutxlm-base")
```

Similar to LayoutLMv2, you can use [`LayoutXLMProcessor`] (which internally applies
[`LayoutLMv2ImageProcessor`] and
[`LayoutXLMTokenizer`]/[`LayoutXLMTokenizerFast`] in sequence) to prepare all
data for the model.

<Tip>

As LayoutXLM's architecture is equivalent to that of LayoutLMv2, one can refer to [LayoutLMv2's documentation page](layoutlmv2) for all tips, code examples and notebooks.
</Tip>

## LayoutXLMTokenizer

[[autodoc]] LayoutXLMTokenizer
    - __call__
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## LayoutXLMTokenizerFast

[[autodoc]] LayoutXLMTokenizerFast
    - __call__

## LayoutXLMProcessor

[[autodoc]] LayoutXLMProcessor
    - __call__


key: _node_content
string_value:
{"id_": "0e2b98ce-c03b-4ad3-b840-a0aed139c8ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "38b2d5c4-18e7-4ba1-96f3-2bdd90de5a83", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "hash": "70e2ff3d7a29b90e95bb444bfb9561e5670d993bd13785a43893145ab2cf7663", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abaf7526-6b06-4994-b17a-9afcce7261e9", "node_type": "1", "metadata": {}, "hash": "cb6d4636a7e2f8a902f0ba299c3550383d1b19657c4f17f6438af1f26a29487c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2853, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MegatronBERT

## Overview

The MegatronBERT model was proposed in [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model
Parallelism](https://arxiv.org/abs/1909.08053) by Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley,
Jared Casper and Bryan Catanzaro.

The abstract from the paper is the following:

*Recent work in language modeling demonstrates that training large transformer models advances the state of the art in
Natural Language Processing applications. However, very large models can be quite difficult to train due to memory
constraints. In this work, we present our techniques for training very large transformer models and implement a simple,
efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our
approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model
parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We
illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain
15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline
that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance
the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9
billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in
BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we
achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA
accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy
of 89.4%).*

This model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). 
That repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, 
it contains a hybrid model parallel approach using "tensor parallel" and "pipeline parallel" techniques.

## Usage tips

We have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints
for use to evaluate or finetuning downstream tasks.

To access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)
Registry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).


key: _node_content
string_value:
{"id_": "abaf7526-6b06-4994-b17a-9afcce7261e9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "38b2d5c4-18e7-4ba1-96f3-2bdd90de5a83", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "hash": "70e2ff3d7a29b90e95bb444bfb9561e5670d993bd13785a43893145ab2cf7663", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e2b98ce-c03b-4ad3-b840-a0aed139c8ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "hash": "d146e6d934b8bbb96048de9cdbc97c858a525477112aed031680e2c7ae88c2e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5ee1ccd-504a-45ea-bad3-985dd2948503", "node_type": "1", "metadata": {}, "hash": "3cdd11a072a95f91e7e95f426ae5b33469743a147584db1713803c99bf23491c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1337, "end_char_idx": 3537, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To demonstrate that large language models can further advance
the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9
billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in
BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we
achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA
accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy
of 89.4%).*

This model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). 
That repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, 
it contains a hybrid model parallel approach using "tensor parallel" and "pipeline parallel" techniques.

## Usage tips

We have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints
for use to evaluate or finetuning downstream tasks.

To access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)
Registry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).

Alternatively, you can directly download the checkpoints using:

BERT-345M-uncased:

```bash
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip
-O megatron_bert_345m_v0_1_uncased.zip
```

BERT-345M-cased:

```bash
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O
megatron_bert_345m_v0_1_cased.zip
```

Once you have obtained the checkpoints from NVIDIA GPU Cloud (NGC), you have to convert them to a format that will
easily be loaded by Hugging Face Transformers and our port of the BERT code.

The following commands allow you to do the conversion.


key: _node_content
string_value:
{"id_": "e5ee1ccd-504a-45ea-bad3-985dd2948503", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "38b2d5c4-18e7-4ba1-96f3-2bdd90de5a83", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "hash": "70e2ff3d7a29b90e95bb444bfb9561e5670d993bd13785a43893145ab2cf7663", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abaf7526-6b06-4994-b17a-9afcce7261e9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers"}, "hash": "fce25c1807d0b9d3d27fc9228a68ccd4ea1a43cf8c7417be381a939d04cd7a71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2071, "end_char_idx": 5249, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). 
That repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, 
it contains a hybrid model parallel approach using "tensor parallel" and "pipeline parallel" techniques.

## Usage tips

We have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints
for use to evaluate or finetuning downstream tasks.

To access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)
Registry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).

Alternatively, you can directly download the checkpoints using:

BERT-345M-uncased:

```bash
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip
-O megatron_bert_345m_v0_1_uncased.zip
```

BERT-345M-cased:

```bash
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O
megatron_bert_345m_v0_1_cased.zip
```

Once you have obtained the checkpoints from NVIDIA GPU Cloud (NGC), you have to convert them to a format that will
easily be loaded by Hugging Face Transformers and our port of the BERT code.

The following commands allow you to do the conversion. We assume that the folder `models/megatron_bert` contains
`megatron_bert_345m_v0_1_{cased, uncased}.zip` and that the commands are run from inside that folder:

```bash
python3 $PATH_TO_TRANSFORMERS/models/megatron_bert/convert_megatron_bert_checkpoint.py megatron_bert_345m_v0_1_uncased.zip
```

```bash
python3 $PATH_TO_TRANSFORMERS/models/megatron_bert/convert_megatron_bert_checkpoint.py megatron_bert_345m_v0_1_cased.zip
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## MegatronBertConfig

[[autodoc]] MegatronBertConfig

## MegatronBertModel

[[autodoc]] MegatronBertModel
    - forward

## MegatronBertForMaskedLM

[[autodoc]] MegatronBertForMaskedLM
    - forward

## MegatronBertForCausalLM

[[autodoc]] MegatronBertForCausalLM
    - forward

## MegatronBertForNextSentencePrediction

[[autodoc]] MegatronBertForNextSentencePrediction
    - forward

## MegatronBertForPreTraining

[[autodoc]] MegatronBertForPreTraining
    - forward

## MegatronBertForSequenceClassification

[[autodoc]] MegatronBertForSequenceClassification
    - forward

## MegatronBertForMultipleChoice

[[autodoc]] MegatronBertForMultipleChoice
    - forward

## MegatronBertForTokenClassification

[[autodoc]] MegatronBertForTokenClassification
    - forward

## MegatronBertForQuestionAnswering

[[autodoc]] MegatronBertForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "3573741f-6037-4967-b293-219d9caa2a9a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc16379d-31ba-4ad7-a399-a6ca57902ae6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers"}, "hash": "61bc83ab28435751bdfd4263c41458251e18c4d000493fe39a77d44cf36f56c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc5872dc-fc96-4301-8800-e30389f6dac4", "node_type": "1", "metadata": {}, "hash": "d20385311eda57a0037cf62219d3e6514f0aafad268755f4b951cdd63ddfa2c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 940, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLM-ProphetNet

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=xprophetnet">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/xprophetnet-large-wiki100-cased-xglue-ntg">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign
@patrickvonplaten


key: _node_content
string_value:
{"id_": "dc5872dc-fc96-4301-8800-e30389f6dac4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fc16379d-31ba-4ad7-a399-a6ca57902ae6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers"}, "hash": "61bc83ab28435751bdfd4263c41458251e18c4d000493fe39a77d44cf36f56c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3573741f-6037-4967-b293-219d9caa2a9a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers"}, "hash": "aaf2fc33b13f2830513c5d739c2015f8e2897f5a79266f6a8445d9c820803959", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3442, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLM-ProphetNet

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=xprophetnet">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/xprophetnet-large-wiki100-cased-xglue-ntg">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>

**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign
@patrickvonplaten


## Overview

The XLM-ProphetNet model was proposed in [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
Zhang, Ming Zhou on 13 Jan, 2020.

XLM-ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of
just the next token. Its architecture is identical to ProhpetNet, but the model was trained on the multi-lingual
"wiki100" Wikipedia dump. XLM-ProphetNet's model architecture and pretraining objective is same as ProphetNet, but XLM-ProphetNet was pre-trained on the cross-lingual dataset XGLUE.

The abstract from the paper is the following:

*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel
self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent
overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
state-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*

The Authors' code can be found [here](https://github.com/microsoft/ProphetNet).

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## XLMProphetNetConfig

[[autodoc]] XLMProphetNetConfig

## XLMProphetNetTokenizer

[[autodoc]] XLMProphetNetTokenizer

## XLMProphetNetModel

[[autodoc]] XLMProphetNetModel

## XLMProphetNetEncoder

[[autodoc]] XLMProphetNetEncoder

## XLMProphetNetDecoder

[[autodoc]] XLMProphetNetDecoder

## XLMProphetNetForConditionalGeneration

[[autodoc]] XLMProphetNetForConditionalGeneration

## XLMProphetNetForCausalLM

[[autodoc]] XLMProphetNetForCausalLM


key: _node_content
string_value:
{"id_": "d061405c-90c6-4e18-86a9-59c96d69afef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/open-llama", "title": "Open-Llama", "tokens": 523, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f10ff20d-dc58-432d-b1fa-c6bce5ff68da", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/open-llama", "title": "Open-Llama", "tokens": 523, "retrieve_doc": true, "source": "transformers"}, "hash": "5942ea5a19e1f1c5703b3a327baecda326badebb76f53fc3451b5b56b7e3d84a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1418, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Open-Llama

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.

If you run into any issues running this model, please reinstall the last version that supported this model: v4.31.0.
You can do so by running the following command: `pip install -U transformers==4.31.0`.

</Tip>

<Tip warning={true}>

This model differs from the [OpenLLaMA models](https://huggingface.co/models?search=openllama) on the Hugging Face Hub, which primarily use the [LLaMA](llama) architecture.

</Tip>

## Overview

The Open-Llama model was proposed in the open source Open-Llama project by community developer s-JoL.

The model is mainly based on LLaMA with some modifications, incorporating memory-efficient attention from Xformers, stable embedding from Bloom, and shared input-output embedding from PaLM.
And the model is pre-trained on both Chinese and English, which gives it better performance on Chinese language tasks.

This model was contributed by [s-JoL](https://huggingface.co/s-JoL).
The original code was released on GitHub by [s-JoL](https://github.com/s-JoL), but is now removed.

## OpenLlamaConfig

[[autodoc]] OpenLlamaConfig

## OpenLlamaModel

[[autodoc]] OpenLlamaModel
    - forward

## OpenLlamaForCausalLM

[[autodoc]] OpenLlamaForCausalLM
    - forward

## OpenLlamaForSequenceClassification

[[autodoc]] OpenLlamaForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "262c647d-314f-4413-ad10-2b27d09f45c7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2cd079dd-247e-411c-a641-cd478f1b9eb4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79687a7e-0a0e-43b7-80ec-c66be9edf689", "node_type": "1", "metadata": {}, "hash": "74284ebc51ce0dc77778d818d996671bd4a235357803d538f2f8e275cfd84d97", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2863, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Phi

## Overview

The Phi-1 model was proposed in [Textbooks Are All You Need](https://arxiv.org/abs/2306.11644) by Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee and Yuanzhi Li.

The Phi-1.5 model was proposed in [Textbooks Are All You Need II: phi-1.5 technical report](https://arxiv.org/abs/2309.05463) by Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar and Yin Tat Lee.

### Summary

In Phi-1 and Phi-1.5 papers, the authors showed how important the quality of the data is in training relative to the model size.
They selected high quality "textbook" data alongside with synthetically generated data for training their small sized Transformer
based model Phi-1 with 1.3B parameters. Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.
They follow the same strategy for Phi-1.5 and created another 1.3B parameter model with performance on natural language tasks comparable
to models 5x larger, and surpassing most non-frontier LLMs. Phi-1.5 exhibits many of the traits of much larger LLMs such as the ability
to “think step by step” or perform some rudimentary in-context learning.
With these two experiments the authors successfully showed the huge impact of quality of training data when training machine learning models.

The abstract from the Phi-1 paper is the following:

*We introduce phi-1, a new large language model for code, with significantly smaller size than
competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on
8 A100s, using a selection of “textbook quality” data from the web (6B tokens) and synthetically
generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent
properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding
exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as
phi-1 that still achieves 45% on HumanEval.*

The abstract from the Phi-1.5 paper is the following:

*We continue the investigation into the power of smaller Transformer-based language models as
initiated by TinyStories – a 10 million parameter model that can produce coherent English – and
the follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close
to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to
generate “textbook quality” data as a way to enhance the learning process compared to traditional
web data.


key: _node_content
string_value:
{"id_": "79687a7e-0a0e-43b7-80ec-c66be9edf689", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2cd079dd-247e-411c-a641-cd478f1b9eb4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "262c647d-314f-4413-ad10-2b27d09f45c7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "1470e3b6fb408a41c406796d2477668d1d9e81f1b301bab0297c8d0ddc1cac30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a478292c-7d25-4f63-adea-3264afcd8353", "node_type": "1", "metadata": {}, "hash": "dc38f22066d503c6187ba9898e841dd2c7173ed4fa013fdfe4680c2fb4924e86", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1251, "end_char_idx": 4372, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Phi-1.5 exhibits many of the traits of much larger LLMs such as the ability
to “think step by step” or perform some rudimentary in-context learning.
With these two experiments the authors successfully showed the huge impact of quality of training data when training machine learning models.

The abstract from the Phi-1 paper is the following:

*We introduce phi-1, a new large language model for code, with significantly smaller size than
competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on
8 A100s, using a selection of “textbook quality” data from the web (6B tokens) and synthetically
generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent
properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding
exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as
phi-1 that still achieves 45% on HumanEval.*

The abstract from the Phi-1.5 paper is the following:

*We continue the investigation into the power of smaller Transformer-based language models as
initiated by TinyStories – a 10 million parameter model that can produce coherent English – and
the follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close
to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to
generate “textbook quality” data as a way to enhance the learning process compared to traditional
web data. We follow the “Textbooks Are All You Need” approach, focusing this time on common
sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5,
with performance on natural language tasks comparable to models 5x larger, and surpassing most
non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic
coding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good –such
as the ability to “think step by step” or perform some rudimentary in-context learning– and bad,
including hallucinations and the potential for toxic and biased generations –encouragingly though, we
are seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to
promote further research on these urgent topics.*

This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).

The original code for Phi-1, Phi-1.5 and Phi-2 can be found [here](https://huggingface.co/microsoft/phi-1), [here](https://huggingface.co/microsoft/phi-1_5) and [here](https://huggingface.co/microsoft/phi-2), respectively.

## Usage tips

- This model is quite similar to `Llama` with the main difference in [`PhiDecoderLayer`], where they used [`PhiAttention`] and [`PhiMLP`] layers in parallel configuration.
- The tokenizer used for this model is identical to the [`CodeGenTokenizer`].

## How to use Phi-2

<Tip warning={true}>

Phi-2 has been integrated in the development version (4.37.0.dev) of `transformers`.


key: _node_content
string_value:
{"id_": "a478292c-7d25-4f63-adea-3264afcd8353", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2cd079dd-247e-411c-a641-cd478f1b9eb4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79687a7e-0a0e-43b7-80ec-c66be9edf689", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "9d3a747800bd86c394bb91ee0c2490916715fa67c9078aa25226d1e51043a359", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bba2b591-3a2e-4c4a-a0d9-1638dfa71f20", "node_type": "1", "metadata": {}, "hash": "638d20c204657d5239234b3165a3b06de54a47d483c94110b44c8a4dbbdffdb7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2864, "end_char_idx": 5975, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We follow the “Textbooks Are All You Need” approach, focusing this time on common
sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5,
with performance on natural language tasks comparable to models 5x larger, and surpassing most
non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic
coding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good –such
as the ability to “think step by step” or perform some rudimentary in-context learning– and bad,
including hallucinations and the potential for toxic and biased generations –encouragingly though, we
are seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to
promote further research on these urgent topics.*

This model was contributed by [Susnato Dhar](https://huggingface.co/susnato).

The original code for Phi-1, Phi-1.5 and Phi-2 can be found [here](https://huggingface.co/microsoft/phi-1), [here](https://huggingface.co/microsoft/phi-1_5) and [here](https://huggingface.co/microsoft/phi-2), respectively.

## Usage tips

- This model is quite similar to `Llama` with the main difference in [`PhiDecoderLayer`], where they used [`PhiAttention`] and [`PhiMLP`] layers in parallel configuration.
- The tokenizer used for this model is identical to the [`CodeGenTokenizer`].

## How to use Phi-2

<Tip warning={true}>

Phi-2 has been integrated in the development version (4.37.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:

* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.

* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.

</Tip>

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")

>>> inputs = tokenizer('Can you help me write a formal email to a potential business partner proposing a joint venture?', return_tensors="pt", return_attention_mask=False)

>>> outputs = model.generate(**inputs, max_length=30)
>>> text = tokenizer.batch_decode(outputs)[0]
>>> print(text)
Can you help me write a formal email to a potential business partner proposing a joint venture?
Input: Company A: ABC Inc.
Company B
```

### Example :

```python
>>> from transformers import PhiForCausalLM, AutoTokenizer

>>> # define the model and tokenizer.
>>> model = PhiForCausalLM.from_pretrained("microsoft/phi-1_5")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5")

>>> # feel free to change the prompt to your liking.
>>> prompt = "If I were an AI that had just achieved"

>>> # apply the tokenizer.
>>> tokens = tokenizer(prompt, return_tensors="pt")

>>> # use the model to generate new tokens.


key: _node_content
string_value:
{"id_": "bba2b591-3a2e-4c4a-a0d9-1638dfa71f20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2cd079dd-247e-411c-a641-cd478f1b9eb4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a478292c-7d25-4f63-adea-3264afcd8353", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "a3c747c1c1e52e24e3b95b010205a24795048b982b06b3081a85f9021692999f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80def581-76d6-40d7-ad90-d73f265f5673", "node_type": "1", "metadata": {}, "hash": "7b1844cd0723baaef3ce814dcfa12aaf633198619363a9910fb7bb4dd8ba8548", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4373, "end_char_idx": 7378, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Until the official version is released through `pip`, ensure that you are doing one of the following:

* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.

* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.

</Tip>

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")

>>> inputs = tokenizer('Can you help me write a formal email to a potential business partner proposing a joint venture?', return_tensors="pt", return_attention_mask=False)

>>> outputs = model.generate(**inputs, max_length=30)
>>> text = tokenizer.batch_decode(outputs)[0]
>>> print(text)
Can you help me write a formal email to a potential business partner proposing a joint venture?
Input: Company A: ABC Inc.
Company B
```

### Example :

```python
>>> from transformers import PhiForCausalLM, AutoTokenizer

>>> # define the model and tokenizer.
>>> model = PhiForCausalLM.from_pretrained("microsoft/phi-1_5")
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5")

>>> # feel free to change the prompt to your liking.
>>> prompt = "If I were an AI that had just achieved"

>>> # apply the tokenizer.
>>> tokens = tokenizer(prompt, return_tensors="pt")

>>> # use the model to generate new tokens.
>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)

>>> tokenizer.batch_decode(generated_output)[0]
'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'
```

## Combining Phi and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import PhiForCausalLM, AutoTokenizer

>>> # define the model and tokenizer and push the model and tokens to the GPU.
>>> model = PhiForCausalLM.from_pretrained("microsoft/phi-1_5", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda")  # doctest: +SKIP
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5")

>>> # feel free to change the prompt to your liking.
>>> prompt = "If I were an AI that had just achieved"

>>> # apply the tokenizer.
>>> tokens = tokenizer(prompt, return_tensors="pt").to("cuda")

>>> # use the model to generate new tokens.


key: _node_content
string_value:
{"id_": "80def581-76d6-40d7-ad90-d73f265f5673", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2cd079dd-247e-411c-a641-cd478f1b9eb4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bba2b591-3a2e-4c4a-a0d9-1638dfa71f20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers"}, "hash": "2206a9704f5f80d5508d4f98fc78ffa23fb6646974598d0d9ebf06a341b4a1ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5797, "end_char_idx": 8444, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> prompt = "If I were an AI that had just achieved"

>>> # apply the tokenizer.
>>> tokens = tokenizer(prompt, return_tensors="pt")

>>> # use the model to generate new tokens.
>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)

>>> tokenizer.batch_decode(generated_output)[0]
'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'
```

## Combining Phi and Flash Attention 2

First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.

```bash
pip install -U flash-attn --no-build-isolation
```

Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)

To load and run a model using Flash Attention 2, refer to the snippet below:

```python
>>> import torch
>>> from transformers import PhiForCausalLM, AutoTokenizer

>>> # define the model and tokenizer and push the model and tokens to the GPU.
>>> model = PhiForCausalLM.from_pretrained("microsoft/phi-1_5", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to("cuda")  # doctest: +SKIP
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5")

>>> # feel free to change the prompt to your liking.
>>> prompt = "If I were an AI that had just achieved"

>>> # apply the tokenizer.
>>> tokens = tokenizer(prompt, return_tensors="pt").to("cuda")

>>> # use the model to generate new tokens.
>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)  # doctest: +SKIP

>>> tokenizer.batch_decode(generated_output)[0]  # doctest: +SKIP
'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'
```

### Expected speedups

Below is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `microsoft/phi-1` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/phi_1_speedup_plot.jpg">
</div>

## PhiConfig

[[autodoc]] PhiConfig

<frameworkcontent>
<pt>

## PhiModel

[[autodoc]] PhiModel
    - forward

## PhiForCausalLM

[[autodoc]] PhiForCausalLM
    - forward
    - generate

## PhiForSequenceClassification

[[autodoc]] PhiForSequenceClassification
    - forward

## PhiForTokenClassification

[[autodoc]] PhiForTokenClassification
    - forward

</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "21df01fc-4a95-4f6b-a7f5-a07dfa7d5121", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c6bcecfc-3adf-4089-b12a-e379b1207f8b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "98463124-209b-44ef-a611-60952e2e2238", "node_type": "1", "metadata": {}, "hash": "b480bec7f264a3e1595308ceeaf69c432724d85b3d574536faab7df3c82178d7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Auto Classes

In many cases, the architecture you want to use can be guessed from the name or the path of the pretrained model you
are supplying to the `from_pretrained()` method. AutoClasses are here to do this job for you so that you
automatically retrieve the relevant model given the name/path to the pretrained weights/config/vocabulary.

Instantiating one of [`AutoConfig`], [`AutoModel`], and
[`AutoTokenizer`] will directly create a class of the relevant architecture. For instance


```python
model = AutoModel.from_pretrained("google-bert/bert-base-cased")
```

will create a model that is an instance of [`BertModel`].

There is one class of `AutoModel` for each task, and for each backend (PyTorch, TensorFlow, or Flax).

## Extending the Auto Classes

Each of the auto classes has a method to be extended with your custom classes. For instance, if you have defined a
custom class of model `NewModel`, make sure you have a `NewModelConfig` then you can add those to the auto
classes like this:

```python
from transformers import AutoConfig, AutoModel

AutoConfig.register("new-model", NewModelConfig)
AutoModel.register(NewModelConfig, NewModel)
```

You will then be able to use the auto classes like you would usually do!

<Tip warning={true}>

If your `NewModelConfig` is a subclass of [`~transformers.PretrainedConfig`], make sure its
`model_type` attribute is set to the same key you use when registering the config (here `"new-model"`).

Likewise, if your `NewModel` is a subclass of [`PreTrainedModel`], make sure its
`config_class` attribute is set to the same class you use when registering the model (here
`NewModelConfig`).

</Tip>

## AutoConfig

[[autodoc]] AutoConfig

## AutoTokenizer

[[autodoc]] AutoTokenizer

## AutoFeatureExtractor

[[autodoc]] AutoFeatureExtractor

## AutoImageProcessor

[[autodoc]] AutoImageProcessor

## AutoProcessor

[[autodoc]] AutoProcessor

## Generic model classes

The following auto classes are available for instantiating a base model class without a specific head.

### AutoModel

[[autodoc]] AutoModel

### TFAutoModel

[[autodoc]] TFAutoModel

### FlaxAutoModel

[[autodoc]] FlaxAutoModel

## Generic pretraining classes

The following auto classes are available for instantiating a model with a pretraining head.

### AutoModelForPreTraining

[[autodoc]] AutoModelForPreTraining

### TFAutoModelForPreTraining

[[autodoc]] TFAutoModelForPreTraining

### FlaxAutoModelForPreTraining

[[autodoc]] FlaxAutoModelForPreTraining

## Natural Language Processing

The following auto classes are available for the following natural language processing tasks.


key: _node_content
string_value:
{"id_": "98463124-209b-44ef-a611-60952e2e2238", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c6bcecfc-3adf-4089-b12a-e379b1207f8b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "21df01fc-4a95-4f6b-a7f5-a07dfa7d5121", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "fbeaee0c0abee71615d5efa921523274b29cc6c27eae97f74a94dbe25d995b75", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f2b80f1-79c8-4ad2-a49a-1847a36ca8e8", "node_type": "1", "metadata": {}, "hash": "30ae3c78bd44c2ef7cfe2313accaa334224d87f03ecaadd1f7793697758e7a77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1240, "end_char_idx": 4875, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip warning={true}>

If your `NewModelConfig` is a subclass of [`~transformers.PretrainedConfig`], make sure its
`model_type` attribute is set to the same key you use when registering the config (here `"new-model"`).

Likewise, if your `NewModel` is a subclass of [`PreTrainedModel`], make sure its
`config_class` attribute is set to the same class you use when registering the model (here
`NewModelConfig`).

</Tip>

## AutoConfig

[[autodoc]] AutoConfig

## AutoTokenizer

[[autodoc]] AutoTokenizer

## AutoFeatureExtractor

[[autodoc]] AutoFeatureExtractor

## AutoImageProcessor

[[autodoc]] AutoImageProcessor

## AutoProcessor

[[autodoc]] AutoProcessor

## Generic model classes

The following auto classes are available for instantiating a base model class without a specific head.

### AutoModel

[[autodoc]] AutoModel

### TFAutoModel

[[autodoc]] TFAutoModel

### FlaxAutoModel

[[autodoc]] FlaxAutoModel

## Generic pretraining classes

The following auto classes are available for instantiating a model with a pretraining head.

### AutoModelForPreTraining

[[autodoc]] AutoModelForPreTraining

### TFAutoModelForPreTraining

[[autodoc]] TFAutoModelForPreTraining

### FlaxAutoModelForPreTraining

[[autodoc]] FlaxAutoModelForPreTraining

## Natural Language Processing

The following auto classes are available for the following natural language processing tasks.

### AutoModelForCausalLM

[[autodoc]] AutoModelForCausalLM

### TFAutoModelForCausalLM

[[autodoc]] TFAutoModelForCausalLM

### FlaxAutoModelForCausalLM

[[autodoc]] FlaxAutoModelForCausalLM

### AutoModelForMaskedLM

[[autodoc]] AutoModelForMaskedLM

### TFAutoModelForMaskedLM

[[autodoc]] TFAutoModelForMaskedLM

### FlaxAutoModelForMaskedLM

[[autodoc]] FlaxAutoModelForMaskedLM

### AutoModelForMaskGeneration

[[autodoc]] AutoModelForMaskGeneration

### TFAutoModelForMaskGeneration

[[autodoc]] TFAutoModelForMaskGeneration

### AutoModelForSeq2SeqLM

[[autodoc]] AutoModelForSeq2SeqLM

### TFAutoModelForSeq2SeqLM

[[autodoc]] TFAutoModelForSeq2SeqLM

### FlaxAutoModelForSeq2SeqLM

[[autodoc]] FlaxAutoModelForSeq2SeqLM

### AutoModelForSequenceClassification

[[autodoc]] AutoModelForSequenceClassification

### TFAutoModelForSequenceClassification

[[autodoc]] TFAutoModelForSequenceClassification

### FlaxAutoModelForSequenceClassification

[[autodoc]] FlaxAutoModelForSequenceClassification

### AutoModelForMultipleChoice

[[autodoc]] AutoModelForMultipleChoice

### TFAutoModelForMultipleChoice

[[autodoc]] TFAutoModelForMultipleChoice

### FlaxAutoModelForMultipleChoice

[[autodoc]] FlaxAutoModelForMultipleChoice

### AutoModelForNextSentencePrediction

[[autodoc]] AutoModelForNextSentencePrediction

### TFAutoModelForNextSentencePrediction

[[autodoc]] TFAutoModelForNextSentencePrediction

### FlaxAutoModelForNextSentencePrediction

[[autodoc]] FlaxAutoModelForNextSentencePrediction

### AutoModelForTokenClassification

[[autodoc]] AutoModelForTokenClassification

### TFAutoModelForTokenClassification

[[autodoc]] TFAutoModelForTokenClassification

### FlaxAutoModelForTokenClassification

[[autodoc]] FlaxAutoModelForTokenClassification

### AutoModelForQuestionAnswering

[[autodoc]] AutoModelForQuestionAnswering

### TFAutoModelForQuestionAnswering

[[autodoc]] TFAutoModelForQuestionAnswering

### FlaxAutoModelForQuestionAnswering

[[autodoc]] FlaxAutoModelForQuestionAnswering

### AutoModelForTextEncoding

[[autodoc]] AutoModelForTextEncoding

### TFAutoModelForTextEncoding

[[autodoc]] TFAutoModelForTextEncoding

## Computer vision

The following auto classes are available for the following computer vision tasks.


key: _node_content
string_value:
{"id_": "6f2b80f1-79c8-4ad2-a49a-1847a36ca8e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c6bcecfc-3adf-4089-b12a-e379b1207f8b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98463124-209b-44ef-a611-60952e2e2238", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "5dfbc35aa8ffb60d9be5e806b8ff4cf18434f991b9198c3cef4c07b72fca79c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe12430d-1482-4714-b66c-cfc492aa726d", "node_type": "1", "metadata": {}, "hash": "483b737e38141b09087203cb31518ffd892e9da53950bfdd4658aacf7413e188", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4877, "end_char_idx": 7318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### AutoModelForDepthEstimation

[[autodoc]] AutoModelForDepthEstimation

### AutoModelForImageClassification

[[autodoc]] AutoModelForImageClassification

### TFAutoModelForImageClassification

[[autodoc]] TFAutoModelForImageClassification

### FlaxAutoModelForImageClassification

[[autodoc]] FlaxAutoModelForImageClassification

### AutoModelForVideoClassification

[[autodoc]] AutoModelForVideoClassification

### AutoModelForKeypointDetection

[[autodoc]] AutoModelForKeypointDetection

### AutoModelForMaskedImageModeling

[[autodoc]] AutoModelForMaskedImageModeling

### TFAutoModelForMaskedImageModeling

[[autodoc]] TFAutoModelForMaskedImageModeling

### AutoModelForObjectDetection

[[autodoc]] AutoModelForObjectDetection

### AutoModelForImageSegmentation

[[autodoc]] AutoModelForImageSegmentation

### AutoModelForImageToImage

[[autodoc]] AutoModelForImageToImage

### AutoModelForSemanticSegmentation

[[autodoc]] AutoModelForSemanticSegmentation

### TFAutoModelForSemanticSegmentation

[[autodoc]] TFAutoModelForSemanticSegmentation

### AutoModelForInstanceSegmentation

[[autodoc]] AutoModelForInstanceSegmentation

### AutoModelForUniversalSegmentation

[[autodoc]] AutoModelForUniversalSegmentation

### AutoModelForZeroShotImageClassification

[[autodoc]] AutoModelForZeroShotImageClassification

### TFAutoModelForZeroShotImageClassification

[[autodoc]] TFAutoModelForZeroShotImageClassification

### AutoModelForZeroShotObjectDetection

[[autodoc]] AutoModelForZeroShotObjectDetection

## Audio

The following auto classes are available for the following audio tasks.

### AutoModelForAudioClassification

[[autodoc]] AutoModelForAudioClassification

### AutoModelForAudioFrameClassification

[[autodoc]] TFAutoModelForAudioClassification

### TFAutoModelForAudioFrameClassification

[[autodoc]] AutoModelForAudioFrameClassification

### AutoModelForCTC

[[autodoc]] AutoModelForCTC

### AutoModelForSpeechSeq2Seq

[[autodoc]] AutoModelForSpeechSeq2Seq

### TFAutoModelForSpeechSeq2Seq

[[autodoc]] TFAutoModelForSpeechSeq2Seq

### FlaxAutoModelForSpeechSeq2Seq

[[autodoc]] FlaxAutoModelForSpeechSeq2Seq

### AutoModelForAudioXVector

[[autodoc]] AutoModelForAudioXVector

### AutoModelForTextToSpectrogram

[[autodoc]] AutoModelForTextToSpectrogram

### AutoModelForTextToWaveform

[[autodoc]] AutoModelForTextToWaveform

## Multimodal

The following auto classes are available for the following multimodal tasks.


key: _node_content
string_value:
{"id_": "fe12430d-1482-4714-b66c-cfc492aa726d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c6bcecfc-3adf-4089-b12a-e379b1207f8b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f2b80f1-79c8-4ad2-a49a-1847a36ca8e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers"}, "hash": "2a424f3a6ad8688527df8c59d0496f74dc902cc29fee1a2f3d4e4f9617a2b22e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6472, "end_char_idx": 7984, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### AutoModelForAudioClassification

[[autodoc]] AutoModelForAudioClassification

### AutoModelForAudioFrameClassification

[[autodoc]] TFAutoModelForAudioClassification

### TFAutoModelForAudioFrameClassification

[[autodoc]] AutoModelForAudioFrameClassification

### AutoModelForCTC

[[autodoc]] AutoModelForCTC

### AutoModelForSpeechSeq2Seq

[[autodoc]] AutoModelForSpeechSeq2Seq

### TFAutoModelForSpeechSeq2Seq

[[autodoc]] TFAutoModelForSpeechSeq2Seq

### FlaxAutoModelForSpeechSeq2Seq

[[autodoc]] FlaxAutoModelForSpeechSeq2Seq

### AutoModelForAudioXVector

[[autodoc]] AutoModelForAudioXVector

### AutoModelForTextToSpectrogram

[[autodoc]] AutoModelForTextToSpectrogram

### AutoModelForTextToWaveform

[[autodoc]] AutoModelForTextToWaveform

## Multimodal

The following auto classes are available for the following multimodal tasks.

### AutoModelForTableQuestionAnswering

[[autodoc]] AutoModelForTableQuestionAnswering

### TFAutoModelForTableQuestionAnswering

[[autodoc]] TFAutoModelForTableQuestionAnswering

### AutoModelForDocumentQuestionAnswering

[[autodoc]] AutoModelForDocumentQuestionAnswering

### TFAutoModelForDocumentQuestionAnswering

[[autodoc]] TFAutoModelForDocumentQuestionAnswering

### AutoModelForVisualQuestionAnswering

[[autodoc]] AutoModelForVisualQuestionAnswering

### AutoModelForVision2Seq

[[autodoc]] AutoModelForVision2Seq

### TFAutoModelForVision2Seq

[[autodoc]] TFAutoModelForVision2Seq

### FlaxAutoModelForVision2Seq

[[autodoc]] FlaxAutoModelForVision2Seq


key: _node_content
string_value:
{"id_": "ae476d5c-e85a-4456-95f2-eb75412fa030", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/instructblip", "title": "InstructBLIP", "tokens": 835, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7c84312-8de1-4078-9f5a-8e2b1ceb4bc6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/instructblip", "title": "InstructBLIP", "tokens": 835, "retrieve_doc": true, "source": "transformers"}, "hash": "8edb96b9117f51e50618d63f1fbc41cc4f982e5820e5158038031b15ec22f7d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3060, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# InstructBLIP

## Overview

The InstructBLIP model was proposed in [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500) by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.
InstructBLIP leverages the [BLIP-2](blip2) architecture for visual instruction tuning.

The abstract from the paper is the following:

*General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/instructblip_architecture.jpg"
alt="drawing" width="600"/>

<small> InstructBLIP architecture. Taken from the <a href="https://arxiv.org/abs/2305.06500">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).

## Usage tips

InstructBLIP uses the same architecture as [BLIP-2](blip2) with a tiny but important difference: it also feeds the text prompt (instruction) to the Q-Former.

## InstructBlipConfig

[[autodoc]] InstructBlipConfig
    - from_vision_qformer_text_configs

## InstructBlipVisionConfig

[[autodoc]] InstructBlipVisionConfig

## InstructBlipQFormerConfig

[[autodoc]] InstructBlipQFormerConfig

## InstructBlipProcessor

[[autodoc]] InstructBlipProcessor


## InstructBlipVisionModel

[[autodoc]] InstructBlipVisionModel
    - forward

## InstructBlipQFormerModel

[[autodoc]] InstructBlipQFormerModel
    - forward

## InstructBlipForConditionalGeneration

[[autodoc]] InstructBlipForConditionalGeneration
    - forward
    - generate


key: _node_content
string_value:
{"id_": "80424c0d-9506-4691-a5f2-1e027edc58a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb0d450f-0606-47a6-93ad-17226fe5a50a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers"}, "hash": "c2cc174a1d7856c89dd0b4ef11ae76fe6155bb64ae49e86b5519453ac5ab3700", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d4de563-3f69-4f01-8f87-6db89e184c0d", "node_type": "1", "metadata": {}, "hash": "52d5ef87898cc8306b9bf41e59d2a3a99819c8436dd87d1095668ccc9a0ad455", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1959, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MPNet

## Overview

The MPNet model was proposed in [MPNet: Masked and Permuted Pre-training for Language Understanding](https://arxiv.org/abs/2004.09297) by Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu.

MPNet adopts a novel pre-training method, named masked and permuted language modeling, to inherit the advantages of
masked language modeling and permuted language modeling for natural language understanding.

The abstract from the paper is the following:

*BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models.
Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for
pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and
thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel
pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the
dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position
information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in
XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of
down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large
margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g.,
BERT, XLNet, RoBERTa) under the same model setting.*

The original code can be found [here](https://github.com/microsoft/MPNet).

## Usage tips

MPNet doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just 
separate your segments with the separation token `tokenizer.sep_token` (or `[sep]`).


key: _node_content
string_value:
{"id_": "7d4de563-3f69-4f01-8f87-6db89e184c0d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb0d450f-0606-47a6-93ad-17226fe5a50a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers"}, "hash": "c2cc174a1d7856c89dd0b4ef11ae76fe6155bb64ae49e86b5519453ac5ab3700", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80424c0d-9506-4691-a5f2-1e027edc58a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers"}, "hash": "e2ceeedf1db052add18d8a07114e18bc5e29d1097773428f446c2c50b77c17fb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 217, "end_char_idx": 3568, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
MPNet adopts a novel pre-training method, named masked and permuted language modeling, to inherit the advantages of
masked language modeling and permuted language modeling for natural language understanding.

The abstract from the paper is the following:

*BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models.
Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for
pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and
thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel
pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the
dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position
information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in
XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of
down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large
margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g.,
BERT, XLNet, RoBERTa) under the same model setting.*

The original code can be found [here](https://github.com/microsoft/MPNet).

## Usage tips

MPNet doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just 
separate your segments with the separation token `tokenizer.sep_token` (or `[sep]`).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## MPNetConfig

[[autodoc]] MPNetConfig

## MPNetTokenizer

[[autodoc]] MPNetTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## MPNetTokenizerFast

[[autodoc]] MPNetTokenizerFast

<frameworkcontent>
<pt>

## MPNetModel

[[autodoc]] MPNetModel
    - forward

## MPNetForMaskedLM

[[autodoc]] MPNetForMaskedLM
    - forward

## MPNetForSequenceClassification

[[autodoc]] MPNetForSequenceClassification
    - forward

## MPNetForMultipleChoice

[[autodoc]] MPNetForMultipleChoice
    - forward

## MPNetForTokenClassification

[[autodoc]] MPNetForTokenClassification
    - forward

## MPNetForQuestionAnswering

[[autodoc]] MPNetForQuestionAnswering
    - forward

</pt>
<tf>

## TFMPNetModel

[[autodoc]] TFMPNetModel
    - call

## TFMPNetForMaskedLM

[[autodoc]] TFMPNetForMaskedLM
    - call

## TFMPNetForSequenceClassification

[[autodoc]] TFMPNetForSequenceClassification
    - call

## TFMPNetForMultipleChoice

[[autodoc]] TFMPNetForMultipleChoice
    - call

## TFMPNetForTokenClassification

[[autodoc]] TFMPNetForTokenClassification
    - call

## TFMPNetForQuestionAnswering

[[autodoc]] TFMPNetForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e33fd2f4-749a-48e6-88ed-20cfc1c2e0d7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b9d0bb89-8644-445a-8613-7b8c6f2aec15", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers"}, "hash": "b7400bbb015bd9b3b1bc661b1c0729a4d1b519c73160f31ba722b6b7fc56c4bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83bcb000-0859-4a0d-8d85-08ca9aa7301e", "node_type": "1", "metadata": {}, "hash": "d4c629fbaf95d990a1d8d6b7a2426a232158392b387882bcc01f1a99fcf3c04d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ConvNeXT

## Overview

The ConvNeXT model was proposed in [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545) by Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie.
ConvNeXT is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, that claims to outperform them.

The abstract from the paper is the following:

*The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model.
A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers
(e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide
variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive
biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually "modernize" a standard ResNet toward the design
of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models
dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy
and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg"
alt="drawing" width="600"/>

<small> ConvNeXT architecture. Taken from the <a href="https://arxiv.org/abs/2201.03545">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [ariG23498](https://github.com/ariG23498),
[gante](https://github.com/gante), and [sayakpaul](https://github.com/sayakpaul) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ConvNeXT.

<PipelineTag pipeline="image-classification"/>

- [`ConvNextForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "83bcb000-0859-4a0d-8d85-08ca9aa7301e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b9d0bb89-8644-445a-8613-7b8c6f2aec15", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers"}, "hash": "b7400bbb015bd9b3b1bc661b1c0729a4d1b519c73160f31ba722b6b7fc56c4bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e33fd2f4-749a-48e6-88ed-20cfc1c2e0d7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers"}, "hash": "dd241be7ac9d4fc423b264baa8a6a15228b6242accc681a5b3de030b85f3f78c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1528, "end_char_idx": 3808, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy
and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg"
alt="drawing" width="600"/>

<small> ConvNeXT architecture. Taken from the <a href="https://arxiv.org/abs/2201.03545">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [ariG23498](https://github.com/ariG23498),
[gante](https://github.com/gante), and [sayakpaul](https://github.com/sayakpaul) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ConvNeXT.

<PipelineTag pipeline="image-classification"/>

- [`ConvNextForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ConvNextConfig

[[autodoc]] ConvNextConfig

## ConvNextFeatureExtractor

[[autodoc]] ConvNextFeatureExtractor

## ConvNextImageProcessor

[[autodoc]] ConvNextImageProcessor
    - preprocess

<frameworkcontent>
<pt>

## ConvNextModel

[[autodoc]] ConvNextModel
    - forward

## ConvNextForImageClassification

[[autodoc]] ConvNextForImageClassification
    - forward

</pt>
<tf>

## TFConvNextModel

[[autodoc]] TFConvNextModel
    - call

## TFConvNextForImageClassification

[[autodoc]] TFConvNextForImageClassification
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "aa668c40-35c3-4c27-8d59-5925b7c284b7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cba45a9-5e68-4319-bc56-13367da1687a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49e8bd05-72f5-4739-aa5b-9255b19205d0", "node_type": "1", "metadata": {}, "hash": "6ef2972ab154af51afa8fa35ef8672b0eedf9815107eb22c4e604abb57e6c1e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3177, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SegFormer

## Overview

The SegFormer model was proposed in [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://arxiv.org/abs/2105.15203) by Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping
Luo. The model consists of a hierarchical Transformer encoder and a lightweight all-MLP decode head to achieve great
results on image segmentation benchmarks such as ADE20K and Cityscapes.

The abstract from the paper is the following:

*We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with
lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel
hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding,
thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution
differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from
different layers, and thus combining both local attention and global attention to render powerful representations. We
show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our
approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance
and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters,
being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on
Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.*

The figure below illustrates the architecture of SegFormer. Taken from the [original paper](https://arxiv.org/abs/2105.15203).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/segformer_architecture.png"/>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version
of the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/NVlabs/SegFormer).

## Usage tips

- SegFormer consists of a hierarchical Transformer encoder, and a lightweight all-MLP decoder head.
  [`SegformerModel`] is the hierarchical Transformer encoder (which in the paper is also referred to
  as Mix Transformer or MiT). [`SegformerForSemanticSegmentation`] adds the all-MLP decoder head on
  top to perform semantic segmentation of images. In addition, there's
  [`SegformerForImageClassification`] which can be used to - you guessed it - classify images. The
  authors of SegFormer first pre-trained the Transformer encoder on ImageNet-1k to classify images. Next, they throw
  away the classification head, and replace it by the all-MLP decode head. Next, they fine-tune the model altogether on
  ADE20K, Cityscapes and COCO-stuff, which are important benchmarks for semantic segmentation. All checkpoints can be
  found on the [hub](https://huggingface.co/models?other=segformer).


key: _node_content
string_value:
{"id_": "49e8bd05-72f5-4739-aa5b-9255b19205d0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cba45a9-5e68-4319-bc56-13367da1687a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa668c40-35c3-4c27-8d59-5925b7c284b7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "1eaeccc05b7340451e3d3aa42c788d5004060bef70ddd63bd012072643dfa167", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6ac6a285-04b9-40b9-a134-731112061537", "node_type": "1", "metadata": {}, "hash": "f336006770b1f40293791d20ff21088930ffa729eb1d39035ab249b25f68baaa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1620, "end_char_idx": 4785, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Our best model, SegFormer-B5, achieves 84.0% mIoU on
Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.*

The figure below illustrates the architecture of SegFormer. Taken from the [original paper](https://arxiv.org/abs/2105.15203).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/segformer_architecture.png"/>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version
of the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/NVlabs/SegFormer).

## Usage tips

- SegFormer consists of a hierarchical Transformer encoder, and a lightweight all-MLP decoder head.
  [`SegformerModel`] is the hierarchical Transformer encoder (which in the paper is also referred to
  as Mix Transformer or MiT). [`SegformerForSemanticSegmentation`] adds the all-MLP decoder head on
  top to perform semantic segmentation of images. In addition, there's
  [`SegformerForImageClassification`] which can be used to - you guessed it - classify images. The
  authors of SegFormer first pre-trained the Transformer encoder on ImageNet-1k to classify images. Next, they throw
  away the classification head, and replace it by the all-MLP decode head. Next, they fine-tune the model altogether on
  ADE20K, Cityscapes and COCO-stuff, which are important benchmarks for semantic segmentation. All checkpoints can be
  found on the [hub](https://huggingface.co/models?other=segformer).
- The quickest way to get started with SegFormer is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer) (which showcase both inference and
  fine-tuning on custom data). One can also check out the [blog post](https://huggingface.co/blog/fine-tune-segformer) introducing SegFormer and illustrating how it can be fine-tuned on custom data.
- TensorFlow users should refer to [this repository](https://github.com/deep-diver/segformer-tf-transformers) that shows off-the-shelf inference and fine-tuning.
- One can also check out [this interactive demo on Hugging Face Spaces](https://huggingface.co/spaces/chansung/segformer-tf-transformers)
  to try out a SegFormer model on custom images.
- SegFormer works on any input size, as it pads the input to be divisible by `config.patch_sizes`.
- One can use [`SegformerImageProcessor`] to prepare images and corresponding segmentation maps
  for the model. Note that this image processor is fairly basic and does not include all data augmentations used in
  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most
  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,
  such as 512x512 or 640x640, after which they are normalized.
- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with
  `do_reduce_labels` set to `True` or `False`.


key: _node_content
string_value:
{"id_": "6ac6a285-04b9-40b9-a134-731112061537", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cba45a9-5e68-4319-bc56-13367da1687a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49e8bd05-72f5-4739-aa5b-9255b19205d0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "4b9311c0177fdf39ba8718abe34d67cb967dded220cc7692bc95672340eb6772", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a79d2a4-8ca2-4ec2-9ee1-d41700823758", "node_type": "1", "metadata": {}, "hash": "46576fe9aea188e6b532d6b6f1a1dae1a076ba0c16b7649ffbaa395e43fef3c8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3178, "end_char_idx": 5659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- The quickest way to get started with SegFormer is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer) (which showcase both inference and
  fine-tuning on custom data). One can also check out the [blog post](https://huggingface.co/blog/fine-tune-segformer) introducing SegFormer and illustrating how it can be fine-tuned on custom data.
- TensorFlow users should refer to [this repository](https://github.com/deep-diver/segformer-tf-transformers) that shows off-the-shelf inference and fine-tuning.
- One can also check out [this interactive demo on Hugging Face Spaces](https://huggingface.co/spaces/chansung/segformer-tf-transformers)
  to try out a SegFormer model on custom images.
- SegFormer works on any input size, as it pads the input to be divisible by `config.patch_sizes`.
- One can use [`SegformerImageProcessor`] to prepare images and corresponding segmentation maps
  for the model. Note that this image processor is fairly basic and does not include all data augmentations used in
  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most
  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,
  such as 512x512 or 640x640, after which they are normalized.
- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with
  `do_reduce_labels` set to `True` or `False`. In some datasets (like ADE20k), the 0 index is used in the annotated
  segmentation maps for background. However, ADE20k doesn't include the "background" class in its 150 labels.
  Therefore, `do_reduce_labels` is used to reduce all labels by 1, and to make sure no loss is computed for the
  background class (i.e. it replaces 0 in the annotated maps by 255, which is the *ignore_index* of the loss function
  used by [`SegformerForSemanticSegmentation`]). However, other datasets use the 0 index as
  background class and include this class as part of all labels. In that case, `do_reduce_labels` should be set to
  `False`, as loss should also be computed for the background class.
- As most models, SegFormer comes in different sizes, the details of which can be found in the table below
  (taken from Table 7 of the [original paper](https://arxiv.org/abs/2105.15203)).


key: _node_content
string_value:
{"id_": "4a79d2a4-8ca2-4ec2-9ee1-d41700823758", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cba45a9-5e68-4319-bc56-13367da1687a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6ac6a285-04b9-40b9-a134-731112061537", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "1a7fc2fd6d781aa5cc5dfb8b6999f5f30821a29999c478895a302090acc9a4d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a98a051-02a2-494a-ab42-9c58fd04cd06", "node_type": "1", "metadata": {}, "hash": "f30e53434db5327c93966eeb77321e6568c73e00f85384acb87057ed9d0547c5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4137, "end_char_idx": 6782, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that this image processor is fairly basic and does not include all data augmentations used in
  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most
  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,
  such as 512x512 or 640x640, after which they are normalized.
- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with
  `do_reduce_labels` set to `True` or `False`. In some datasets (like ADE20k), the 0 index is used in the annotated
  segmentation maps for background. However, ADE20k doesn't include the "background" class in its 150 labels.
  Therefore, `do_reduce_labels` is used to reduce all labels by 1, and to make sure no loss is computed for the
  background class (i.e. it replaces 0 in the annotated maps by 255, which is the *ignore_index* of the loss function
  used by [`SegformerForSemanticSegmentation`]). However, other datasets use the 0 index as
  background class and include this class as part of all labels. In that case, `do_reduce_labels` should be set to
  `False`, as loss should also be computed for the background class.
- As most models, SegFormer comes in different sizes, the details of which can be found in the table below
  (taken from Table 7 of the [original paper](https://arxiv.org/abs/2105.15203)).

| **Model variant** | **Depths**    | **Hidden sizes**    | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |
| :---------------: | ------------- | ------------------- | :---------------------: | :------------: | :-------------------: |
| MiT-b0            | [2, 2, 2, 2]  | [32, 64, 160, 256]  | 256                     | 3.7            | 70.5                  |
| MiT-b1            | [2, 2, 2, 2]  | [64, 128, 320, 512] | 256                     | 14.0           | 78.7                  |
| MiT-b2            | [3, 4, 6, 3]  | [64, 128, 320, 512] | 768                     | 25.4           | 81.6                  |
| MiT-b3            | [3, 4, 18, 3] | [64, 128, 320, 512] | 768                     | 45.2           | 83.1                  |
| MiT-b4            | [3, 8, 27, 3] | [64, 128, 320, 512] | 768                     | 62.6           | 83.6                  |
| MiT-b5            | [3, 6, 40, 3] | [64, 128, 320, 512] | 768                     | 82.0           | 83.8                  |

Note that MiT in the above table refers to the Mix Transformer encoder backbone introduced in SegFormer.


key: _node_content
string_value:
{"id_": "0a98a051-02a2-494a-ab42-9c58fd04cd06", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cba45a9-5e68-4319-bc56-13367da1687a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a79d2a4-8ca2-4ec2-9ee1-d41700823758", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "0e6fbe4f9d9b0b0785321383926f805a26b103243b97686fc6e1dcdaa61e33c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c1ee0690-917f-411b-9c2a-a232d43e2280", "node_type": "1", "metadata": {}, "hash": "dc0f3945eafb535d4a7deb531a0249224ee1918e045a2b964b3483e6a5c1beb8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5661, "end_char_idx": 8418, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
| **Model variant** | **Depths**    | **Hidden sizes**    | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |
| :---------------: | ------------- | ------------------- | :---------------------: | :------------: | :-------------------: |
| MiT-b0            | [2, 2, 2, 2]  | [32, 64, 160, 256]  | 256                     | 3.7            | 70.5                  |
| MiT-b1            | [2, 2, 2, 2]  | [64, 128, 320, 512] | 256                     | 14.0           | 78.7                  |
| MiT-b2            | [3, 4, 6, 3]  | [64, 128, 320, 512] | 768                     | 25.4           | 81.6                  |
| MiT-b3            | [3, 4, 18, 3] | [64, 128, 320, 512] | 768                     | 45.2           | 83.1                  |
| MiT-b4            | [3, 8, 27, 3] | [64, 128, 320, 512] | 768                     | 62.6           | 83.6                  |
| MiT-b5            | [3, 6, 40, 3] | [64, 128, 320, 512] | 768                     | 82.0           | 83.8                  |

Note that MiT in the above table refers to the Mix Transformer encoder backbone introduced in SegFormer. For
SegFormer's results on the segmentation datasets like ADE20k, refer to the [paper](https://arxiv.org/abs/2105.15203).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SegFormer.

<PipelineTag pipeline="image-classification"/>

- [`SegformerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- [Image classification task guide](../tasks/image_classification)

Semantic segmentation:

- [`SegformerForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation).
- A blog on fine-tuning SegFormer on a custom dataset can be found [here](https://huggingface.co/blog/fine-tune-segformer).
- More demo notebooks on SegFormer (both inference + fine-tuning on a custom dataset) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).
- [`TFSegformerForSemanticSegmentation`] is supported by this [example notebook](https://github.com/huggingface/notebooks/blob/main/examples/semantic_segmentation-tf.ipynb).
- [Semantic segmentation task guide](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "c1ee0690-917f-411b-9c2a-a232d43e2280", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cba45a9-5e68-4319-bc56-13367da1687a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a98a051-02a2-494a-ab42-9c58fd04cd06", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers"}, "hash": "a4d784ef7c4d4ac1745d3e42a69370a904e8919575e1fe98a40d8eb65efeb8f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6783, "end_char_idx": 9431, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For
SegFormer's results on the segmentation datasets like ADE20k, refer to the [paper](https://arxiv.org/abs/2105.15203).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with SegFormer.

<PipelineTag pipeline="image-classification"/>

- [`SegformerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- [Image classification task guide](../tasks/image_classification)

Semantic segmentation:

- [`SegformerForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation).
- A blog on fine-tuning SegFormer on a custom dataset can be found [here](https://huggingface.co/blog/fine-tune-segformer).
- More demo notebooks on SegFormer (both inference + fine-tuning on a custom dataset) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).
- [`TFSegformerForSemanticSegmentation`] is supported by this [example notebook](https://github.com/huggingface/notebooks/blob/main/examples/semantic_segmentation-tf.ipynb).
- [Semantic segmentation task guide](../tasks/semantic_segmentation)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## SegformerConfig

[[autodoc]] SegformerConfig

## SegformerFeatureExtractor

[[autodoc]] SegformerFeatureExtractor
    - __call__
    - post_process_semantic_segmentation

## SegformerImageProcessor

[[autodoc]] SegformerImageProcessor
    - preprocess
    - post_process_semantic_segmentation

<frameworkcontent>
<pt>

## SegformerModel

[[autodoc]] SegformerModel
    - forward

## SegformerDecodeHead

[[autodoc]] SegformerDecodeHead
    - forward

## SegformerForImageClassification

[[autodoc]] SegformerForImageClassification
    - forward

## SegformerForSemanticSegmentation

[[autodoc]] SegformerForSemanticSegmentation
    - forward

</pt>
<tf>

## TFSegformerDecodeHead

[[autodoc]] TFSegformerDecodeHead
    - call

## TFSegformerModel

[[autodoc]] TFSegformerModel
    - call

## TFSegformerForImageClassification

[[autodoc]] TFSegformerForImageClassification
    - call

## TFSegformerForSemanticSegmentation

[[autodoc]] TFSegformerForSemanticSegmentation
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "43e20aaa-17a8-41e4-b577-82566431bbb6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xls_r", "title": "XLS-R", "tokens": 705, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0c5c26c6-96d1-4175-9079-9930aea5efc4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xls_r", "title": "XLS-R", "tokens": 705, "retrieve_doc": true, "source": "transformers"}, "hash": "b6a3b7c5d6f8afedad8afb749f0acaf2bd55b9787da7c18949f944f4bb293ad9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2170, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLS-R

## Overview

The XLS-R model was proposed in [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://arxiv.org/abs/2111.09296) by Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman
Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli.

The abstract from the paper is the following:

*This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0.
We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128
languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range
of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation
benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into
English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as
VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107
language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform
English-only pretraining when translating English speech into other languages, a setting which favors monolingual
pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.*

Relevant checkpoints can be found under https://huggingface.co/models?other=xls_r.

The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).

## Usage tips

- XLS-R is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- XLS-R model was trained using connectionist temporal classification (CTC) so the model output has to be decoded using
  [`Wav2Vec2CTCTokenizer`].

<Tip>

XLS-R's architecture is based on the Wav2Vec2 model, refer to [Wav2Vec2's documentation page](wav2vec2) for API reference.

</Tip>


key: _node_content
string_value:
{"id_": "a446ecdf-1734-48d2-add3-fcdb15906a61", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-v", "title": "XLM-V", "tokens": 706, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4fdad278-3b6c-499e-b39a-af6cf3f8ed4e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-v", "title": "XLM-V", "tokens": 706, "retrieve_doc": true, "source": "transformers"}, "hash": "c72cdce69276b484c8bd8986ad08c800c44a050a05119c20f640a6f1d7368bb4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLM-V

## Overview

XLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R).
It was introduced in the [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://arxiv.org/abs/2301.10472)
paper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa.

From the abstract of the XLM-V paper:

*Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.
As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged.
This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R.
In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by
de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity
to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically
more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V,
a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we
tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and
named entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).*

This model was contributed by [stefan-it](https://huggingface.co/stefan-it), including detailed experiments with XLM-V on downstream tasks.
The experiments repository can be found [here](https://github.com/stefan-it/xlm-v-experiments).

## Usage tips

- XLM-V is compatible with the XLM-RoBERTa model architecture, only model weights from [`fairseq`](https://github.com/facebookresearch/fairseq)
  library had to be converted.
- The `XLMTokenizer` implementation is used to load the vocab and performs tokenization.

A XLM-V (base size) model is available under the [`facebook/xlm-v-base`](https://huggingface.co/facebook/xlm-v-base) identifier.

<Tip>

XLM-V architecture is the same as XLM-RoBERTa, refer to [XLM-RoBERTa documentation](xlm-roberta) for API reference, and examples.
</Tip>


key: _node_content
string_value:
{"id_": "7553a7ea-5599-4ffe-bffe-25e0a5400b73", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "68c909ad-76a4-462b-8f1f-994dfc9f4481", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers"}, "hash": "b8fd130c73c9d2f7f023e2ceaac967ac46fe27725a3288b49c1ab37d1308c6f8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "34ca539b-12db-41a1-a6c7-d9d5d4398372", "node_type": "1", "metadata": {}, "hash": "c8e97a119d7ccfd449e3f9e46958d008acf9378dc4ebd41c8edab5cf1b3063ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1894, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RemBERT

## Overview

The RemBERT model was proposed in [Rethinking Embedding Coupling in Pre-trained Language Models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, Sebastian Ruder.

The abstract from the paper is the following:

*We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art
pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to
significantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By
reallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on
standard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that
allocating additional capacity to the output embedding provides benefits to the model that persist through the
fine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger
output embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage
Transformer representations to be more general and more transferable to other tasks and languages. Harnessing these
findings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the
number of parameters at the fine-tuning stage.*

## Usage tips

For fine-tuning, RemBERT can be thought of as a bigger version of mBERT with an ALBERT-like factorization of the
embedding layer. The embeddings are not tied in pre-training, in contrast with BERT, which enables smaller input
embeddings (preserved during fine-tuning) and bigger output embeddings (discarded at fine-tuning). The tokenizer is
also similar to the Albert one rather than the BERT one.


key: _node_content
string_value:
{"id_": "34ca539b-12db-41a1-a6c7-d9d5d4398372", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "68c909ad-76a4-462b-8f1f-994dfc9f4481", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers"}, "hash": "b8fd130c73c9d2f7f023e2ceaac967ac46fe27725a3288b49c1ab37d1308c6f8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7553a7ea-5599-4ffe-bffe-25e0a5400b73", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers"}, "hash": "a89c5ae4a57e10ec6fcaac34b18a33bee80086f2713a2d6bc3b0cbd1afef8ed9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3904, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RemBERT

## Overview

The RemBERT model was proposed in [Rethinking Embedding Coupling in Pre-trained Language Models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, Sebastian Ruder.

The abstract from the paper is the following:

*We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art
pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to
significantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By
reallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on
standard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that
allocating additional capacity to the output embedding provides benefits to the model that persist through the
fine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger
output embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage
Transformer representations to be more general and more transferable to other tasks and languages. Harnessing these
findings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the
number of parameters at the fine-tuning stage.*

## Usage tips

For fine-tuning, RemBERT can be thought of as a bigger version of mBERT with an ALBERT-like factorization of the
embedding layer. The embeddings are not tied in pre-training, in contrast with BERT, which enables smaller input
embeddings (preserved during fine-tuning) and bigger output embeddings (discarded at fine-tuning). The tokenizer is
also similar to the Albert one rather than the BERT one.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## RemBertConfig

[[autodoc]] RemBertConfig

## RemBertTokenizer

[[autodoc]] RemBertTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## RemBertTokenizerFast

[[autodoc]] RemBertTokenizerFast
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

<frameworkcontent>
<pt>

## RemBertModel

[[autodoc]] RemBertModel
    - forward

## RemBertForCausalLM

[[autodoc]] RemBertForCausalLM
    - forward

## RemBertForMaskedLM

[[autodoc]] RemBertForMaskedLM
    - forward

## RemBertForSequenceClassification

[[autodoc]] RemBertForSequenceClassification
    - forward

## RemBertForMultipleChoice

[[autodoc]] RemBertForMultipleChoice
    - forward

## RemBertForTokenClassification

[[autodoc]] RemBertForTokenClassification
    - forward

## RemBertForQuestionAnswering

[[autodoc]] RemBertForQuestionAnswering
    - forward

</pt>
<tf>

## TFRemBertModel

[[autodoc]] TFRemBertModel
    - call

## TFRemBertForMaskedLM

[[autodoc]] TFRemBertForMaskedLM
    - call

## TFRemBertForCausalLM

[[autodoc]] TFRemBertForCausalLM
    - call

## TFRemBertForSequenceClassification

[[autodoc]] TFRemBertForSequenceClassification
    - call

## TFRemBertForMultipleChoice

[[autodoc]] TFRemBertForMultipleChoice
    - call

## TFRemBertForTokenClassification

[[autodoc]] TFRemBertForTokenClassification
    - call

## TFRemBertForQuestionAnswering

[[autodoc]] TFRemBertForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "c12de767-54b8-43c1-963c-d36fefcd0253", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_moe", "title": "Qwen2MoE", "tokens": 941, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ceaacfc8-3d0f-4b9c-8e40-6150786b0406", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_moe", "title": "Qwen2MoE", "tokens": 941, "retrieve_doc": true, "source": "transformers"}, "hash": "e5ae81ea9610210ac4b74678145129ff55c1a67580f0f1aac667046df0cd8f76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2797, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Qwen2MoE

## Overview

Qwen2MoE is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen-72B, Qwen-1.8B, Qwen-VL, Qwen-Audio, etc.

### Model Details

Qwen2MoE is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. Qwen2MoE has the following architectural choices:

- Qwen2MoE is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.
- Qwen2MoE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, `Qwen1.5-MoE-A2.7B` is upcycled from `Qwen-1.8B`. It has 14.3B parameters in total and 2.7B activated parameters during runtime, while it achieves comparable performance with `Qwen1.5-7B`, with only 25% of the training resources.

For more details refer to the [release blog post](https://qwenlm.github.io/blog/qwen-moe/).

## Usage tips

`Qwen1.5-MoE-A2.7B` and `Qwen1.5-MoE-A2.7B-Chat` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)

In the following, we demonstrate how to use `Qwen1.5-MoE-A2.7B-Chat` for the inference. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> device = "cuda" # the device to load the model onto

>>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen1.5-MoE-A2.7B-Chat", device_map="auto")
>>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen1.5-MoE-A2.7B-Chat")

>>> prompt = "Give me a short introduction to large language model."

>>> messages = [{"role": "user", "content": prompt}]

>>> text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

>>> model_inputs = tokenizer([text], return_tensors="pt").to(device)

>>> generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)

>>> generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]

>>> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

## Qwen2MoeConfig

[[autodoc]] Qwen2MoeConfig

## Qwen2MoeModel

[[autodoc]] Qwen2MoeModel
    - forward

## Qwen2MoeForCausalLM

[[autodoc]] Qwen2MoeForCausalLM
    - forward

## Qwen2MoeForSequenceClassification

[[autodoc]] Qwen2MoeForSequenceClassification
    - forward

## Qwen2MoeForTokenClassification

[[autodoc]] Qwen2MoeForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "4a283020-a4f3-476e-b367-70785fd0517a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e06b0641-6e46-495d-a81f-9f63158e64f2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a9d21db-1787-4413-8a00-ce908f21e7dd", "node_type": "1", "metadata": {}, "hash": "c545c881b354c5d2123665c77a39fa2246c915604548056bcb996a838e31c2ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3388, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SeamlessM4T

## Overview

The SeamlessM4T model was proposed in [SeamlessM4T — Massively Multilingual & Multimodal Machine Translation](https://dl.fbaipublicfiles.com/seamless/seamless_m4t_paper.pdf) by the Seamless Communication team from Meta AI.

This is the **version 1** release of the model. For the updated **version 2** release, refer to the [Seamless M4T v2 docs](https://huggingface.co/docs/transformers/main/model_doc/seamless_m4t_v2).

SeamlessM4T is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text.

SeamlessM4T enables multiple tasks without relying on separate models:

- Speech-to-speech translation (S2ST)
- Speech-to-text translation (S2TT)
- Text-to-speech translation (T2ST)
- Text-to-text translation (T2TT)
- Automatic speech recognition (ASR)

[`SeamlessM4TModel`] can perform all the above tasks, but each task also has its own dedicated sub-model.

The abstract from the paper is the following:

*What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication*

## Usage

First, load the processor and a checkpoint of the model:

```python
>>> from transformers import AutoProcessor, SeamlessM4TModel

>>> processor = AutoProcessor.from_pretrained("facebook/hf-seamless-m4t-medium")
>>> model = SeamlessM4TModel.from_pretrained("facebook/hf-seamless-m4t-medium")
```

You can seamlessly use this model on text or on audio, to generated either translated text or translated audio.


key: _node_content
string_value:
{"id_": "7a9d21db-1787-4413-8a00-ce908f21e7dd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e06b0641-6e46-495d-a81f-9f63158e64f2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a283020-a4f3-476e-b367-70785fd0517a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "bdd724ea97dcaeb9a9d5d05751c5ee132863d46e1d6db18d2d6bd536d86edb44", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9477f71-69e9-40e9-9902-354fca4930b9", "node_type": "1", "metadata": {}, "hash": "f3fd281d257beb3f8ad6c171d794c992afe58c24c2b5508bb87f8d03d8c5c086", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1800, "end_char_idx": 5077, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication*

## Usage

First, load the processor and a checkpoint of the model:

```python
>>> from transformers import AutoProcessor, SeamlessM4TModel

>>> processor = AutoProcessor.from_pretrained("facebook/hf-seamless-m4t-medium")
>>> model = SeamlessM4TModel.from_pretrained("facebook/hf-seamless-m4t-medium")
```

You can seamlessly use this model on text or on audio, to generated either translated text or translated audio.

Here is how to use the processor to process text and audio:

```python
>>> # let's load an audio sample from an Arabic speech corpus
>>> from datasets import load_dataset
>>> dataset = load_dataset("arabic_speech_corpus", split="test", streaming=True)
>>> audio_sample = next(iter(dataset))["audio"]

>>> # now, process it
>>> audio_inputs = processor(audios=audio_sample["array"], return_tensors="pt")

>>> # now, process some English test as well
>>> text_inputs = processor(text = "Hello, my dog is cute", src_lang="eng", return_tensors="pt")
```


### Speech

[`SeamlessM4TModel`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:

```python
>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
```

With basically the same code, I've translated English text and Arabic speech to Russian speech samples.

### Text

Similarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4TModel.generate`].
This time, let's translate to French.

```python 
>>> # from audio
>>> output_tokens = model.generate(**audio_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)

>>> # from text
>>> output_tokens = model.generate(**text_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)
```

### Tips


key: _node_content
string_value:
{"id_": "e9477f71-69e9-40e9-9902-354fca4930b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e06b0641-6e46-495d-a81f-9f63158e64f2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7a9d21db-1787-4413-8a00-ce908f21e7dd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "0886e82448a2f79aefe6bb08baf388af3dd2fabd9915c278bf29ceebbb47c686", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ded8930-239f-4836-a376-b500e00d74d2", "node_type": "1", "metadata": {}, "hash": "0baafe24a31e2b0559b9a0369a6c30203f6eef4fb92f3b5d0e655aed494f848b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3390, "end_char_idx": 6745, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here is how to use the processor to process text and audio:

```python
>>> # let's load an audio sample from an Arabic speech corpus
>>> from datasets import load_dataset
>>> dataset = load_dataset("arabic_speech_corpus", split="test", streaming=True)
>>> audio_sample = next(iter(dataset))["audio"]

>>> # now, process it
>>> audio_inputs = processor(audios=audio_sample["array"], return_tensors="pt")

>>> # now, process some English test as well
>>> text_inputs = processor(text = "Hello, my dog is cute", src_lang="eng", return_tensors="pt")
```


### Speech

[`SeamlessM4TModel`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:

```python
>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang="rus")[0].cpu().numpy().squeeze()
```

With basically the same code, I've translated English text and Arabic speech to Russian speech samples.

### Text

Similarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4TModel.generate`].
This time, let's translate to French.

```python 
>>> # from audio
>>> output_tokens = model.generate(**audio_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)

>>> # from text
>>> output_tokens = model.generate(**text_inputs, tgt_lang="fra", generate_speech=False)
>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)
```

### Tips


#### 1. Use dedicated models

[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.
For example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: 

```python
>>> from transformers import SeamlessM4TForSpeechToSpeech
>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained("facebook/hf-seamless-m4t-medium")
```

Or you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.

```python
>>> from transformers import SeamlessM4TForTextToText
>>> model = SeamlessM4TForTextToText.from_pretrained("facebook/hf-seamless-m4t-medium")
```

Feel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.

#### 2. Change the speaker identity

You have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!

#### 3. Change the generation strategy

You can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.

#### 4. Generate speech and text at the same time

Use `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !

## Model architecture


key: _node_content
string_value:
{"id_": "5ded8930-239f-4836-a376-b500e00d74d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e06b0641-6e46-495d-a81f-9f63158e64f2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9477f71-69e9-40e9-9902-354fca4930b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "04cd0b58a8c6e7420b2f7a58fac9c04580b657134a11a87f9972c15bad3499b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ac76f8c-6053-4441-83f5-7b4ab45a3449", "node_type": "1", "metadata": {}, "hash": "31224a1abbd7ba94ccb28d25bb81b1aeacfa787a9a29eafa2a9f43532e08eae4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5080, "end_char_idx": 8138, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### 1. Use dedicated models

[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.
For example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: 

```python
>>> from transformers import SeamlessM4TForSpeechToSpeech
>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained("facebook/hf-seamless-m4t-medium")
```

Or you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.

```python
>>> from transformers import SeamlessM4TForTextToText
>>> model = SeamlessM4TForTextToText.from_pretrained("facebook/hf-seamless-m4t-medium")
```

Feel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.

#### 2. Change the speaker identity

You have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!

#### 3. Change the generation strategy

You can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.

#### 4. Generate speech and text at the same time

Use `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !

## Model architecture


SeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as "unit tokens," from the translated text.

Each modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.

Here's how the generation process works:

- Input text or speech is processed through its specific encoder.
- A decoder creates text tokens in the desired language.
- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.
- These unit tokens are then passed through the final vocoder to produce the actual speech.


This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## SeamlessM4TModel

[[autodoc]] SeamlessM4TModel
    - generate


## SeamlessM4TForTextToSpeech

[[autodoc]] SeamlessM4TForTextToSpeech
    - generate


## SeamlessM4TForSpeechToSpeech

[[autodoc]] SeamlessM4TForSpeechToSpeech
    - generate


key: _node_content
string_value:
{"id_": "9ac76f8c-6053-4441-83f5-7b4ab45a3449", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e06b0641-6e46-495d-a81f-9f63158e64f2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ded8930-239f-4836-a376-b500e00d74d2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers"}, "hash": "9eddd8099b9a468e15b8f2880e09c4e48e91914d4e9dd81ae516d215df6507a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6748, "end_char_idx": 9167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
SeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as "unit tokens," from the translated text.

Each modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.

Here's how the generation process works:

- Input text or speech is processed through its specific encoder.
- A decoder creates text tokens in the desired language.
- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.
- These unit tokens are then passed through the final vocoder to produce the actual speech.


This model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).

## SeamlessM4TModel

[[autodoc]] SeamlessM4TModel
    - generate


## SeamlessM4TForTextToSpeech

[[autodoc]] SeamlessM4TForTextToSpeech
    - generate


## SeamlessM4TForSpeechToSpeech

[[autodoc]] SeamlessM4TForSpeechToSpeech
    - generate


## SeamlessM4TForTextToText

[[autodoc]] transformers.SeamlessM4TForTextToText
    - forward
    - generate

## SeamlessM4TForSpeechToText

[[autodoc]] transformers.SeamlessM4TForSpeechToText
    - forward
    - generate

## SeamlessM4TConfig

[[autodoc]] SeamlessM4TConfig


## SeamlessM4TTokenizer

[[autodoc]] SeamlessM4TTokenizer
    - __call__
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary


## SeamlessM4TTokenizerFast

[[autodoc]] SeamlessM4TTokenizerFast
    - __call__

## SeamlessM4TFeatureExtractor

[[autodoc]] SeamlessM4TFeatureExtractor
    - __call__

## SeamlessM4TProcessor

[[autodoc]] SeamlessM4TProcessor
    - __call__

## SeamlessM4TCodeHifiGan

[[autodoc]] SeamlessM4TCodeHifiGan


## SeamlessM4THifiGan

[[autodoc]] SeamlessM4THifiGan

## SeamlessM4TTextToUnitModel

[[autodoc]] SeamlessM4TTextToUnitModel

## SeamlessM4TTextToUnitForConditionalGeneration

[[autodoc]] SeamlessM4TTextToUnitForConditionalGeneration


key: _node_content
string_value:
{"id_": "a090822a-c34a-495c-8d53-6c61bf74bdaa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "beb2607c-ec49-4165-9b95-0ecf7db48056", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "17a29695-38b3-4f7e-a4fa-1aeda1d10984", "node_type": "1", "metadata": {}, "hash": "e20dafc9d6623ee78a309cc42aabc93d29c085b753dc87c9290905ac1c24940e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ImageGPT

## Overview

The ImageGPT model was proposed in [Generative Pretraining from Pixels](https://openai.com/blog/image-gpt) by Mark
Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever. ImageGPT (iGPT) is a GPT-2-like
model trained to predict the next pixel value, allowing for both unconditional and conditional image generation.

The abstract from the paper is the following:

*Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models
can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels,
without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels,
we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and
low-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide
ResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pre-trained models. We are also
competitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0%
top-1 accuracy on a linear probe of our features.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/imagegpt_architecture.png"
alt="drawing" width="600"/>

<small> Summary of the approach. Taken from the [original paper](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf). </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr), based on [this issue](https://github.com/openai/image-gpt/issues/7). The original code can be found
[here](https://github.com/openai/image-gpt).

## Usage tips

- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation
  function is used (namely "quick gelu"), and the layer normalization layers don't mean center the inputs. ImageGPT
  also doesn't have tied input- and output embeddings.
- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence
  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special "start of sentence" (SOS)
  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare
  images for the model.
- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
  performant image features useful for downstream tasks, such as image classification.


key: _node_content
string_value:
{"id_": "17a29695-38b3-4f7e-a4fa-1aeda1d10984", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "beb2607c-ec49-4165-9b95-0ecf7db48056", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a090822a-c34a-495c-8d53-6c61bf74bdaa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "aae7680741c4708254757654b747601a97226a4022ea008b338cc09ad6e89a87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1182a63b-3a38-4fb6-9878-c35fd1d7bc2f", "node_type": "1", "metadata": {}, "hash": "5fccfa84855df62b8f1903509f9f52776a67d488581354f7e837cf743acf608c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1717, "end_char_idx": 3883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code can be found
[here](https://github.com/openai/image-gpt).

## Usage tips

- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation
  function is used (namely "quick gelu"), and the layer normalization layers don't mean center the inputs. ImageGPT
  also doesn't have tied input- and output embeddings.
- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence
  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special "start of sentence" (SOS)
  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare
  images for the model.
- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
  performant image features useful for downstream tasks, such as image classification. The authors showed that the
  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as
  a sklearn logistic regression model for example). This is also referred to as "linear probing". Features can be
  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and
  then average-pool the hidden states at whatever layer you like.
- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can
  use [`ImageGPTForImageClassification`].
- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also
  train an XL variant, which they didn't release.


key: _node_content
string_value:
{"id_": "1182a63b-3a38-4fb6-9878-c35fd1d7bc2f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "beb2607c-ec49-4165-9b95-0ecf7db48056", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17a29695-38b3-4f7e-a4fa-1aeda1d10984", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "87ae26fec03f36364a0103118620ba77e642585fa3c05ec71d4dab4087cbc87d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03bf357c-5c56-4d82-b78d-f8eabcfc08dd", "node_type": "1", "metadata": {}, "hash": "42c5f4faa6c1ce043a0454ff6de7536d5d55d600c806f46e56d7872e218dedbd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2300, "end_char_idx": 4630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, feeding a
  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors
  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long
  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger
  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special "start of sentence" (SOS)
  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare
  images for the model.
- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly
  performant image features useful for downstream tasks, such as image classification. The authors showed that the
  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as
  a sklearn logistic regression model for example). This is also referred to as "linear probing". Features can be
  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and
  then average-pool the hidden states at whatever layer you like.
- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can
  use [`ImageGPTForImageClassification`].
- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also
  train an XL variant, which they didn't release. The differences in size are summarized in the following table:

| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |
|---|---|---|---|---|---|
| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |
| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |
| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |
| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |
| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |
| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ImageGPT.


key: _node_content
string_value:
{"id_": "03bf357c-5c56-4d82-b78d-f8eabcfc08dd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "beb2607c-ec49-4165-9b95-0ecf7db48056", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1182a63b-3a38-4fb6-9878-c35fd1d7bc2f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers"}, "hash": "36d1f8384f88c5f14538bfcda8cd4c7da2b2cf340657789a70d889f9cac5e465", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3813, "end_char_idx": 5858, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The authors did also
  train an XL variant, which they didn't release. The differences in size are summarized in the following table:

| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |
|---|---|---|---|---|---|
| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |
| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |
| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |
| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |
| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |
| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ImageGPT.

<PipelineTag pipeline="image-classification"/>

- Demo notebooks for ImageGPT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ImageGPT).
- [`ImageGPTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## ImageGPTConfig

[[autodoc]] ImageGPTConfig

## ImageGPTFeatureExtractor

[[autodoc]] ImageGPTFeatureExtractor
    - __call__

## ImageGPTImageProcessor

[[autodoc]] ImageGPTImageProcessor
    - preprocess

## ImageGPTModel

[[autodoc]] ImageGPTModel
    - forward

## ImageGPTForCausalImageModeling

[[autodoc]] ImageGPTForCausalImageModeling
    - forward

## ImageGPTForImageClassification

[[autodoc]] ImageGPTForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "9a27d880-6468-427f-b705-8987fd907049", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nezha", "title": "Nezha", "tokens": 857, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ccaee26d-fa1d-4d50-974e-3d15598cfadd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/nezha", "title": "Nezha", "tokens": 857, "retrieve_doc": true, "source": "transformers"}, "hash": "6a0df10602610c8043bee5f55759a8a85a3880618b232e72aff1a48e57aecf4f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2821, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Nezha

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The Nezha model was proposed in [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://arxiv.org/abs/1909.00204) by Junqiu Wei et al.

The abstract from the paper is the following:

*The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks
due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.
In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed
representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks.
The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional
Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy,
Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA
achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including
named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti)
and natural language inference (XNLI).*

This model was contributed by [sijunhe](https://huggingface.co/sijunhe). The original code can be found [here](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-PyTorch).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## NezhaConfig

[[autodoc]] NezhaConfig

## NezhaModel

[[autodoc]] NezhaModel
    - forward

## NezhaForPreTraining

[[autodoc]] NezhaForPreTraining
    - forward

## NezhaForMaskedLM

[[autodoc]] NezhaForMaskedLM
    - forward

## NezhaForNextSentencePrediction

[[autodoc]] NezhaForNextSentencePrediction
    - forward

## NezhaForSequenceClassification

[[autodoc]] NezhaForSequenceClassification
    - forward

## NezhaForMultipleChoice

[[autodoc]] NezhaForMultipleChoice
    - forward

## NezhaForTokenClassification

[[autodoc]] NezhaForTokenClassification
    - forward

## NezhaForQuestionAnswering

[[autodoc]] NezhaForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "cb141594-bbee-40b5-8d6d-a13a0fd37ed9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5eb6f636-530d-49c7-8a13-cd11a53c4c68", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers"}, "hash": "4537d0bafecaafc19090fdc6ad100dad53e1e7b9abcfbbd5e6eeb3cdefb480b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a1de531-0121-4fc4-9acf-d3849c5ffed1", "node_type": "1", "metadata": {}, "hash": "d4e4b6db29d7225bf740abda8c406d28516077dde573d58a7868566c3b43b491", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Audio Spectrogram Transformer

## Overview

The Audio Spectrogram Transformer model was proposed in [AST: Audio Spectrogram Transformer](https://arxiv.org/abs/2104.01778) by Yuan Gong, Yu-An Chung, James Glass.
The Audio Spectrogram Transformer applies a [Vision Transformer](vit) to audio, by turning audio into an image (spectrogram). The model obtains state-of-the-art results
for audio classification.

The abstract from the paper is the following:

*In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png"
alt="drawing" width="600"/>

<small> Audio Spectrogram Transformer architecture. Taken from the <a href="https://arxiv.org/abs/2104.01778">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/YuanGongND/ast).

## Usage tips

- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make
sure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet
mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how
the authors compute the stats for a downstream dataset.
- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the
[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.


key: _node_content
string_value:
{"id_": "5a1de531-0121-4fc4-9acf-d3849c5ffed1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5eb6f636-530d-49c7-8a13-cd11a53c4c68", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers"}, "hash": "4537d0bafecaafc19090fdc6ad100dad53e1e7b9abcfbbd5e6eeb3cdefb480b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cb141594-bbee-40b5-8d6d-a13a0fd37ed9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers"}, "hash": "7e0ec700be214cd923d500fe7e3eece9a42069d784f7d0d590fe7e4c24967883", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0942d78d-16be-4f55-883f-126683e742aa", "node_type": "1", "metadata": {}, "hash": "d901af1d1cdc7253b73f04dc6ef84ddb1b30a0d78d7c3f3bc7d5c81caeba8116", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1651, "end_char_idx": 4891, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2104.01778">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/YuanGongND/ast).

## Usage tips

- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make
sure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet
mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how
the authors compute the stats for a downstream dataset.
- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the
[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ASTForAudioClassification
model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        27 |                                         6 |                      4.5 |
|            2 |                                        12 |                                         6 |                      2   |
|            4 |                                        21 |                                         8 |                      2.62 |
|            8 |                                        40 |                                        14 |                      2.86 |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with the Audio Spectrogram Transformer.


