# embedding_metadata

key: chroma:document
string_value:
# SpeechT5

## Overview

The SpeechT5 model was proposed in [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://arxiv.org/abs/2110.07205) by Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei.

The abstract from the paper is the following:

*Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.*

This model was contributed by [Matthijs](https://huggingface.co/Matthijs). The original code can be found [here](https://github.com/microsoft/SpeechT5).

## SpeechT5Config

[[autodoc]] SpeechT5Config

## SpeechT5HifiGanConfig

[[autodoc]] SpeechT5HifiGanConfig

## SpeechT5Tokenizer

[[autodoc]] SpeechT5Tokenizer
    - __call__
    - save_vocabulary
    - decode
    - batch_decode

## SpeechT5FeatureExtractor

[[autodoc]] SpeechT5FeatureExtractor
    - __call__

## SpeechT5Processor

[[autodoc]] SpeechT5Processor
    - __call__
    - pad
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## SpeechT5Model

[[autodoc]] SpeechT5Model
    - forward

## SpeechT5ForSpeechToText

[[autodoc]] SpeechT5ForSpeechToText
    - forward

## SpeechT5ForTextToSpeech

[[autodoc]] SpeechT5ForTextToSpeech
    - forward
    - generate

## SpeechT5ForSpeechToSpeech

[[autodoc]] SpeechT5ForSpeechToSpeech
    - forward
    - generate_speech

## SpeechT5HifiGan

[[autodoc]] SpeechT5HifiGan
    - forward


key: _node_content
string_value:
{"id_": "9c20daa7-ce47-4383-b90c-dc1246b2ea0b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "67ea26be-073f-493f-af1f-cc539db15387", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers"}, "hash": "cc756dcef81023f96529c34925be84d96b335510f0e706d4887bbf897364c5f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d1c709ed-90c6-4597-998b-5e1c9b3fc48e", "node_type": "1", "metadata": {}, "hash": "e723db98fb83ed78eabc5540a51f436096ac48e25bc0f039e2c90155d197212c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3373, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MobileNet V1

## Overview

The MobileNet model was proposed in [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://arxiv.org/abs/1704.04861) by Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam.

The abstract from the paper is the following:

*We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.*

This model was contributed by [matthijs](https://huggingface.co/Matthijs). The original code and weights can be found [here](https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md).

## Usage tips

- The checkpoints are named **mobilenet\_v1\_*depth*\_*size***, for example **mobilenet\_v1\_1.0\_224**, where **1.0** is the depth multiplier (sometimes also referred to as "alpha" or the width multiplier) and **224** is the resolution of the input images the model was trained on.

- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.

- One can use [`MobileNetV1ImageProcessor`] to prepare images for the model.

- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra “background” class (index 0).

- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV1Config`] with `tf_padding = False`.

Unsupported features:

- The [`MobileNetV1Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use a 7x7 average pooling layer with stride 2 instead of global pooling. For larger inputs, this gives a pooled output that is larger than 1x1 pixel. The HuggingFace implementation does not support this.

- It is currently not possible to specify an `output_stride`. For smaller output strides, the original model invokes dilated convolution to prevent the spatial resolution from being reduced further. The output stride of the HuggingFace model is always 32.

- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional "FakeQuantization" operations to unquantize the weights.


key: _node_content
string_value:
{"id_": "d1c709ed-90c6-4597-998b-5e1c9b3fc48e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "67ea26be-073f-493f-af1f-cc539db15387", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers"}, "hash": "cc756dcef81023f96529c34925be84d96b335510f0e706d4887bbf897364c5f5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9c20daa7-ce47-4383-b90c-dc1246b2ea0b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilenet_v1", "title": "MobileNet V1", "tokens": 1239, "retrieve_doc": true, "source": "transformers"}, "hash": "cefc6fa4e0b6711b01ab6a5a93328d435f2e1a89adb533173b5215a8a54ea80d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1721, "end_char_idx": 4803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Even though the checkpoint is trained on images of specific size, the model will work on images of any size. The smallest supported image size is 32x32.

- One can use [`MobileNetV1ImageProcessor`] to prepare images for the model.

- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes). However, the model predicts 1001 classes: the 1000 classes from ImageNet plus an extra “background” class (index 0).

- The original TensorFlow checkpoints use different padding rules than PyTorch, requiring the model to determine the padding amount at inference time, since this depends on the input image size. To use native PyTorch padding behavior, create a [`MobileNetV1Config`] with `tf_padding = False`.

Unsupported features:

- The [`MobileNetV1Model`] outputs a globally pooled version of the last hidden state. In the original model it is possible to use a 7x7 average pooling layer with stride 2 instead of global pooling. For larger inputs, this gives a pooled output that is larger than 1x1 pixel. The HuggingFace implementation does not support this.

- It is currently not possible to specify an `output_stride`. For smaller output strides, the original model invokes dilated convolution to prevent the spatial resolution from being reduced further. The output stride of the HuggingFace model is always 32.

- The original TensorFlow checkpoints include quantized models. We do not support these models as they include additional "FakeQuantization" operations to unquantize the weights.

- It's common to extract the output from the pointwise layers at indices 5, 11, 12, 13 for downstream purposes. Using `output_hidden_states=True` returns the output from all intermediate layers. There is currently no way to limit this to specific layers.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with MobileNetV1.

<PipelineTag pipeline="image-classification"/>

- [`MobileNetV1ForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## MobileNetV1Config

[[autodoc]] MobileNetV1Config

## MobileNetV1FeatureExtractor

[[autodoc]] MobileNetV1FeatureExtractor
    - preprocess

## MobileNetV1ImageProcessor

[[autodoc]] MobileNetV1ImageProcessor
    - preprocess

## MobileNetV1Model

[[autodoc]] MobileNetV1Model
    - forward

## MobileNetV1ForImageClassification

[[autodoc]] MobileNetV1ForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "c23570ec-19dc-45ef-9c7d-21eac25a5056", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jetmoe", "title": "JetMoe", "tokens": 506, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e663ecc6-63c8-43da-8b5b-7343951fe91b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jetmoe", "title": "JetMoe", "tokens": 506, "retrieve_doc": true, "source": "transformers"}, "hash": "8684ce766325fedc198ac2435f441b62b98fbbe99672150e91a13eadec9131e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1247, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# JetMoe

## Overview

**JetMoe-8B** is an 8B Mixture-of-Experts (MoE) language model developed by [Yikang Shen](https://scholar.google.com.hk/citations?user=qff5rRYAAAAJ) and [MyShell](https://myshell.ai/).
JetMoe project aims to provide a LLaMA2-level performance and efficient language model with a limited budget.
To achieve this goal, JetMoe uses a sparsely activated architecture inspired by the [ModuleFormer](https://arxiv.org/abs/2306.04640). 
Each JetMoe block consists of two MoE layers: Mixture of Attention Heads and Mixture of MLP Experts.
Given the input tokens, it activates a subset of its experts to process them.
This sparse activation schema enables JetMoe to achieve much better training throughput than similar size dense models. 
The training throughput of JetMoe-8B is around 100B tokens per day on a cluster of 96 H100 GPUs with a straightforward 3-way pipeline parallelism strategy.

This model was contributed by [Yikang Shen](https://huggingface.co/YikangS).


## JetMoeConfig

[[autodoc]] JetMoeConfig

## JetMoeModel

[[autodoc]] JetMoeModel
    - forward

## JetMoeForCausalLM

[[autodoc]] JetMoeForCausalLM
    - forward

## JetMoeForSequenceClassification

[[autodoc]] JetMoeForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "130be32a-a792-445a-a8a3-341259e943d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl", "title": "XLM-RoBERTa-XL", "tokens": 849, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "21260625-3704-4554-8734-a32316a5e0bc", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlm-roberta-xl", "title": "XLM-RoBERTa-XL", "tokens": 849, "retrieve_doc": true, "source": "transformers"}, "hash": "75d481267b5776122b84e5856634c7c7bf373394d2081b29254048efaa23d3fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2614, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLM-RoBERTa-XL

## Overview

The XLM-RoBERTa-XL model was proposed in [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://arxiv.org/abs/2105.00572) by Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau. 

The abstract from the paper is the following:

*Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.*

This model was contributed by [Soonhwan-Kwon](https://github.com/Soonhwan-Kwon) and [stefan-it](https://huggingface.co/stefan-it). The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/xlmr).

## Usage tips

XLM-RoBERTa-XL is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does 
not require `lang` tensors to understand which language is used, and should be able to determine the correct 
language from the input ids.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## XLMRobertaXLConfig

[[autodoc]] XLMRobertaXLConfig

## XLMRobertaXLModel

[[autodoc]] XLMRobertaXLModel
    - forward

## XLMRobertaXLForCausalLM

[[autodoc]] XLMRobertaXLForCausalLM
    - forward

## XLMRobertaXLForMaskedLM

[[autodoc]] XLMRobertaXLForMaskedLM
    - forward

## XLMRobertaXLForSequenceClassification

[[autodoc]] XLMRobertaXLForSequenceClassification
    - forward

## XLMRobertaXLForMultipleChoice

[[autodoc]] XLMRobertaXLForMultipleChoice
    - forward

## XLMRobertaXLForTokenClassification

[[autodoc]] XLMRobertaXLForTokenClassification
    - forward

## XLMRobertaXLForQuestionAnswering

[[autodoc]] XLMRobertaXLForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "13a05da0-1552-4b78-aaf9-8cd923c32ecb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bit", "title": "Big Transfer (BiT)", "tokens": 890, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f99d93cc-3e91-479b-aa2a-9fd1f1181d8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bit", "title": "Big Transfer (BiT)", "tokens": 890, "retrieve_doc": true, "source": "transformers"}, "hash": "528ac4260d29d9fc92230c49b6f5fa694d0a5405a764fb143540ad9638c0a137", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3082, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Big Transfer (BiT)

## Overview

The BiT model was proposed in [Big Transfer (BiT): General Visual Representation Learning](https://arxiv.org/abs/1912.11370) by Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby.
BiT is a simple recipe for scaling up pre-training of [ResNet](resnet)-like architectures (specifically, ResNetv2). The method results in significant improvements for transfer learning.

The abstract from the paper is the following:

*Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.*

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/google-research/big_transfer).

## Usage tips

- BiT models are equivalent to ResNetv2 in terms of architecture, except that: 1) all batch normalization layers are replaced by [group normalization](https://arxiv.org/abs/1803.08494),
2) [weight standardization](https://arxiv.org/abs/1903.10520) is used for convolutional layers. The authors show that the combination of both is useful for training with large batch sizes, and has a significant
impact on transfer learning.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with BiT.

<PipelineTag pipeline="image-classification"/>

- [`BitForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).
- See also: [Image classification task guide](../tasks/image_classification)

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## BitConfig

[[autodoc]] BitConfig

## BitImageProcessor

[[autodoc]] BitImageProcessor
    - preprocess

## BitModel

[[autodoc]] BitModel
    - forward

## BitForImageClassification

[[autodoc]] BitForImageClassification
    - forward


key: _node_content
string_value:
{"id_": "54a533e4-b846-49d9-8b84-6199f3ffbaa8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics", "title": "IDEFICS", "tokens": 743, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f2115022-d44b-41db-bae9-35d11628efd3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/idefics", "title": "IDEFICS", "tokens": 743, "retrieve_doc": true, "source": "transformers"}, "hash": "b941ca0f6bc20ee29516c8e0810f9f8fe26d387aeec4c12f95043ab212cf8a90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2319, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# IDEFICS

## Overview

The IDEFICS model was proposed in [OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
](https://huggingface.co/papers/2306.16527
) by Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, Victor Sanh

The abstract from the paper is the following:

*Large multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks that require reasoning over one or multiple images to generate a text. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELISC, we train an 80 billion parameters vision and language model on the dataset and obtain competitive performance on various multimodal benchmarks. We release the code to reproduce the dataset along with the dataset itself.*

This model was contributed by [HuggingFaceM4](https://huggingface.co/HuggingFaceM4). The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>). (TODO: don't have a public link yet).


<Tip warning={true}>

IDEFICS modeling code in Transformers is for finetuning and inferencing the pre-trained IDEFICS models.

To train a new IDEFICS model from scratch use the m4 codebase (a link will be provided once it's made public)

</Tip>


## IdeficsConfig

[[autodoc]] IdeficsConfig

## IdeficsModel

[[autodoc]] IdeficsModel
    - forward

## IdeficsForVisionText2Text

[[autodoc]] IdeficsForVisionText2Text
    - forward

## TFIdeficsModel

[[autodoc]] TFIdeficsModel
    - call

## TFIdeficsForVisionText2Text

[[autodoc]] TFIdeficsForVisionText2Text
    - call

## IdeficsImageProcessor

[[autodoc]] IdeficsImageProcessor
    - preprocess

## IdeficsProcessor

[[autodoc]] IdeficsProcessor
    - __call__


key: _node_content
string_value:
{"id_": "7b5916b6-3faf-4c90-a5cf-22c4eec3d77d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers"}, "hash": "40866bcab7c8c567834d5846571dc84f5d1ba53fb55cb6326439e5d5d4c628b8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d6e4f7e-b04b-49f9-a160-a3e15511804e", "node_type": "1", "metadata": {}, "hash": "e1bdb4b6cbed24efffabdc6e056d3eadf630777174a9729feb5d5d62dbead28d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2977, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ViLT

## Overview

The ViLT model was proposed in [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334)
by Wonjae Kim, Bokyung Son, Ildoo Kim. ViLT incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design
for Vision-and-Language Pre-training (VLP).

The abstract from the paper is the following:

*Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.
Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision
(e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we
find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more
computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive
power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model,
Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically
simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of
times faster than previous VLP models, yet with competitive or better downstream task performance.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg"
alt="drawing" width="600"/>

<small> ViLT architecture. Taken from the <a href="https://arxiv.org/abs/2102.03334">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).

## Usage tips

- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)
  (which showcase both inference and fine-tuning on custom data).
- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.
  This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.
- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to
  under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates
  which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.
- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes
  additional embedding layers for the language modality.
- The PyTorch version of this model is only available in torch 1.10 and higher.


key: _node_content
string_value:
{"id_": "0d6e4f7e-b04b-49f9-a160-a3e15511804e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ab6afca4-f9ff-4791-a08c-bef6083949d9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers"}, "hash": "40866bcab7c8c567834d5846571dc84f5d1ba53fb55cb6326439e5d5d4c628b8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b5916b6-3faf-4c90-a5cf-22c4eec3d77d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vilt", "title": "ViLT", "tokens": 1055, "retrieve_doc": true, "source": "transformers"}, "hash": "f0412cf16d1609611eada350244e6b16fcb969917e09179cde87e6c2b2535e18", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1442, "end_char_idx": 3692, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg"
alt="drawing" width="600"/>

<small> ViLT architecture. Taken from the <a href="https://arxiv.org/abs/2102.03334">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).

## Usage tips

- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)
  (which showcase both inference and fine-tuning on custom data).
- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.
  This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.
- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to
  under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates
  which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.
- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes
  additional embedding layers for the language modality.
- The PyTorch version of this model is only available in torch 1.10 and higher.

## ViltConfig

[[autodoc]] ViltConfig

## ViltFeatureExtractor

[[autodoc]] ViltFeatureExtractor
    - __call__

## ViltImageProcessor

[[autodoc]] ViltImageProcessor
    - preprocess

## ViltProcessor

[[autodoc]] ViltProcessor
    - __call__

## ViltModel

[[autodoc]] ViltModel
    - forward

## ViltForMaskedLM

[[autodoc]] ViltForMaskedLM
    - forward

## ViltForQuestionAnswering

[[autodoc]] ViltForQuestionAnswering
    - forward

## ViltForImagesAndTextClassification

[[autodoc]] ViltForImagesAndTextClassification
    - forward

## ViltForImageAndTextRetrieval

[[autodoc]] ViltForImageAndTextRetrieval
    - forward

## ViltForTokenClassification

[[autodoc]] ViltForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "0b3ac0d3-f761-437c-acec-e1a82fc26c92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpt", "title": "MPT", "tokens": 704, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "354ef785-1919-4196-87f1-00426df9d9a5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mpt", "title": "MPT", "tokens": 704, "retrieve_doc": true, "source": "transformers"}, "hash": "09a8105580481f1d98ee1bebfc0a5025d8e3363e4fc5f2d2ed794ccf4693b168", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2103, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MPT

## Overview

The MPT model was proposed by the [MosaicML](https://www.mosaicml.com/) team and released with multiple sizes and finetuned variants. The MPT models is a series of open source and commercially usable LLMs pre-trained on 1T tokens. 

MPT models are GPT-style decoder-only transformers with several improvements: performance-optimized layer implementations, architecture changes that provide greater training stability, and the elimination of context length limits by replacing positional embeddings with ALiBi. 

- MPT base: MPT base pre-trained models on next token prediction 
- MPT instruct: MPT base models fine-tuned on instruction based tasks
- MPT storywriter: MPT base models fine-tuned for 2500 steps on 65k-token excerpts of fiction books contained in the books3 corpus, this enables the model to handle very long sequences

The original code is available at the  [`llm-foundry`](https://github.com/mosaicml/llm-foundry/tree/main) repository.

Read more about it [in the release blogpost](https://www.mosaicml.com/blog/mpt-7b)

## Usage tips

- Learn more about some techniques behind training of the model [in this section of llm-foundry repository](https://github.com/mosaicml/llm-foundry/blob/main/TUTORIAL.md#faqs)
- If you want to use the advanced version of the model (triton kernels, direct flash attention integration), you can still use the original model implementation by adding `trust_remote_code=True` when calling `from_pretrained`.

## Resources

- [Fine-tuning Notebook](https://colab.research.google.com/drive/1HCpQkLL7UXW8xJUJJ29X7QAeNJKO0frZ?usp=sharing) on how to fine-tune MPT-7B on a free Google Colab instance to turn the model into a Chatbot.

## MptConfig

[[autodoc]] MptConfig
    - all

## MptModel

[[autodoc]] MptModel
    - forward

## MptForCausalLM

[[autodoc]] MptForCausalLM
    - forward

## MptForSequenceClassification

[[autodoc]] MptForSequenceClassification
    - forward

## MptForTokenClassification

[[autodoc]] MptForTokenClassification
    - forward

## MptForQuestionAnswering

[[autodoc]] MptForQuestionAnswering
    - forward


key: _node_content
string_value:
{"id_": "10181ac3-6fb5-489f-b8fe-a254b4bb386b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb41233b-83da-45a6-8f44-ed52193202be", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b0b77bd-506c-41c3-9bfd-1c876520e34a", "node_type": "1", "metadata": {}, "hash": "452619de1cc79c67d6bba72b39f7e7e5b71310d035d82bcd183bb01345f026ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2959, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CodeLlama

## Overview

The Code Llama model was proposed in [Code Llama: Open Foundation Models for Code](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/) by Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve.

The abstract from the paper is the following:

*We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*

Check out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [Meta Llama org](https://huggingface.co/meta-llama).

This model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).

## Usage tips and examples

<Tip warning={true}>

The `Llama2` family models, on which Code Llama is based, were trained using `bfloat16`, but the original inference uses `float16`. Let's look at the different precisions:

* `float32`: PyTorch convention on model initialization is to load models in `float32`, no matter with which `dtype` the model weights were stored. `transformers` also follows this convention for consistency with PyTorch. This will be picked by default. If you want the `AutoModel` API to cast the load the checkpoints with the storage weights type, you must specify `torch_dtype="auto"`, e.g. `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`.
* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.


key: _node_content
string_value:
{"id_": "4b0b77bd-506c-41c3-9bfd-1c876520e34a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb41233b-83da-45a6-8f44-ed52193202be", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10181ac3-6fb5-489f-b8fe-a254b4bb386b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "05dea4d9a0290fa0a20acc15d36be3089515b889f97f151e7aabcb4848741c7c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e13f30e-981b-4b8a-a69f-ba9f87fecf90", "node_type": "1", "metadata": {}, "hash": "2b1cbc272f1079ada86ccd7aac3e504c05ce46c243096b3f18662ae8dedf4bea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1351, "end_char_idx": 4366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.*

Check out all Code Llama model checkpoints [here](https://huggingface.co/models?search=code_llama) and the officially released ones in the [Meta Llama org](https://huggingface.co/meta-llama).

This model was contributed by [ArthurZucker](https://huggingface.co/ArthurZ). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).

## Usage tips and examples

<Tip warning={true}>

The `Llama2` family models, on which Code Llama is based, were trained using `bfloat16`, but the original inference uses `float16`. Let's look at the different precisions:

* `float32`: PyTorch convention on model initialization is to load models in `float32`, no matter with which `dtype` the model weights were stored. `transformers` also follows this convention for consistency with PyTorch. This will be picked by default. If you want the `AutoModel` API to cast the load the checkpoints with the storage weights type, you must specify `torch_dtype="auto"`, e.g. `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`.
* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.
* `float16`: We recommend running inference using this precision, as it's usually faster than `bfloat16`, and evaluation metrics show no discernible degradation with respect to `bfloat16`. You can also run inference using `bfloat16`, and we recommend you check inference results with both `float16` and `bfloat16` after fine-tuning.

As mentioned above, the `dtype` of the storage weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using. The reason is that the model will first be downloaded (using the `dtype` of the checkpoints online) and then will be casted to the default `dtype` of `torch` (becomes `torch.float32`). If there is a specified `torch_dtype`, it will be used instead.

</Tip>


Tips:
- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.
- The model conversion script is the same as for the `Llama2` family:

Here is a sample usage:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).


key: _node_content
string_value:
{"id_": "7e13f30e-981b-4b8a-a69f-ba9f87fecf90", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb41233b-83da-45a6-8f44-ed52193202be", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b0b77bd-506c-41c3-9bfd-1c876520e34a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "7879a7a667e001ded38d968292d0944c7b5cefa03e5fcaa9d22087c3c75c1ca8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a0069c3d-91b0-495b-8aa7-09586f940bfe", "node_type": "1", "metadata": {}, "hash": "70b84ecc4635d909ea9f23b1152a872de5e2f65d3f7be9a81ae610f9e06dabf4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2762, "end_char_idx": 5313, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`.
* `bfloat16`: Code Llama was trained with this precision, so we recommend using it for further training or fine-tuning.
* `float16`: We recommend running inference using this precision, as it's usually faster than `bfloat16`, and evaluation metrics show no discernible degradation with respect to `bfloat16`. You can also run inference using `bfloat16`, and we recommend you check inference results with both `float16` and `bfloat16` after fine-tuning.

As mentioned above, the `dtype` of the storage weights is mostly irrelevant unless you are using `torch_dtype="auto"` when initializing a model using. The reason is that the model will first be downloaded (using the `dtype` of the checkpoints online) and then will be casted to the default `dtype` of `torch` (becomes `torch.float32`). If there is a specified `torch_dtype`, it will be used instead.

</Tip>


Tips:
- The infilling task is supported out of the box. You should be using the `tokenizer.fill_token` where you want your input to be filled.
- The model conversion script is the same as for the `Llama2` family:

Here is a sample usage:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).

After conversion, the model and tokenizer can be loaded via:

```python
>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer

>>> tokenizer = CodeLlamaTokenizer.from_pretrained("meta-llama/CodeLlama-7b-hf")
>>> model = LlamaForCausalLM.from_pretrained("meta-llama/CodeLlama-7b-hf")
>>> PROMPT = '''def remove_non_ascii(s: str) -> str:
...     """ <FILL_ME>
...     return result
... '''
>>> input_ids = tokenizer(PROMPT, return_tensors="pt")["input_ids"]
>>> generated_ids = model.generate(input_ids, max_new_tokens=128)

>>> filling = tokenizer.batch_decode(generated_ids[:, input_ids.shape[1]:], skip_special_tokens = True)[0]
>>> print(PROMPT.replace("<FILL_ME>", filling))
def remove_non_ascii(s: str) -> str:
    """ Remove non-ASCII characters from a string.
<BLANKLINE>
    Args:
        s: The string to remove non-ASCII characters from.
<BLANKLINE>
    Returns:
        The string with non-ASCII characters removed.
    """


key: _node_content
string_value:
{"id_": "a0069c3d-91b0-495b-8aa7-09586f940bfe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb41233b-83da-45a6-8f44-ed52193202be", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7e13f30e-981b-4b8a-a69f-ba9f87fecf90", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "5f3b11954fe9015fa8bf82004d64912aa4c46366e5285e171a0a1260b3cdd146", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26c554f5-a3c6-43c4-9cd7-2cbb71839f29", "node_type": "1", "metadata": {}, "hash": "43b31fe80bd05c5d1117fee998a00415aa7ffac84947db0921e6be6a5d070dda", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3760, "end_char_idx": 6705, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You should be using the `tokenizer.fill_token` where you want your input to be filled.
- The model conversion script is the same as for the `Llama2` family:

Here is a sample usage:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).

After conversion, the model and tokenizer can be loaded via:

```python
>>> from transformers import LlamaForCausalLM, CodeLlamaTokenizer

>>> tokenizer = CodeLlamaTokenizer.from_pretrained("meta-llama/CodeLlama-7b-hf")
>>> model = LlamaForCausalLM.from_pretrained("meta-llama/CodeLlama-7b-hf")
>>> PROMPT = '''def remove_non_ascii(s: str) -> str:
...     """ <FILL_ME>
...     return result
... '''
>>> input_ids = tokenizer(PROMPT, return_tensors="pt")["input_ids"]
>>> generated_ids = model.generate(input_ids, max_new_tokens=128)

>>> filling = tokenizer.batch_decode(generated_ids[:, input_ids.shape[1]:], skip_special_tokens = True)[0]
>>> print(PROMPT.replace("<FILL_ME>", filling))
def remove_non_ascii(s: str) -> str:
    """ Remove non-ASCII characters from a string.
<BLANKLINE>
    Args:
        s: The string to remove non-ASCII characters from.
<BLANKLINE>
    Returns:
        The string with non-ASCII characters removed.
    """
    result = ""
    for c in s:
        if ord(c) < 128:
            result += c
    return result
<BLANKLINE>
```

If you only want the infilled part:
```python
>>> from transformers import pipeline
>>> import torch

>>> generator = pipeline("text-generation",model="meta-llama/CodeLlama-7b-hf",torch_dtype=torch.float16, device_map="auto")
>>> generator('def remove_non_ascii(s: str) -> str:\n    """ <FILL_ME>\n    return result', max_new_tokens = 128)
[{'generated_text': 'def remove_non_ascii(s: str) -> str:\n    """ <FILL_ME>\n    return resultRemove non-ASCII characters from a string. """\n    result = ""\n    for c in s:\n        if ord(c) < 128:\n            result += c'}]
```

Under the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug.  To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value.


key: _node_content
string_value:
{"id_": "26c554f5-a3c6-43c4-9cd7-2cbb71839f29", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cb41233b-83da-45a6-8f44-ed52193202be", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "18affe0a10ce1571f11526a8082f64dd55465a1884ca565bcc3d19dd4e0db30d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a0069c3d-91b0-495b-8aa7-09586f940bfe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/code_llama", "title": "CodeLlama", "tokens": 2064, "retrieve_doc": true, "source": "transformers"}, "hash": "c89154f1cc60f4a2223be658921c0ff3abb012b07e802b4f3b4eddf14253c661", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5227, "end_char_idx": 7598, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<BLANKLINE>
    Returns:
        The string with non-ASCII characters removed.
    """
    result = ""
    for c in s:
        if ord(c) < 128:
            result += c
    return result
<BLANKLINE>
```

If you only want the infilled part:
```python
>>> from transformers import pipeline
>>> import torch

>>> generator = pipeline("text-generation",model="meta-llama/CodeLlama-7b-hf",torch_dtype=torch.float16, device_map="auto")
>>> generator('def remove_non_ascii(s: str) -> str:\n    """ <FILL_ME>\n    return result', max_new_tokens = 128)
[{'generated_text': 'def remove_non_ascii(s: str) -> str:\n    """ <FILL_ME>\n    return resultRemove non-ASCII characters from a string. """\n    result = ""\n    for c in s:\n        if ord(c) < 128:\n            result += c'}]
```

Under the hood, the tokenizer [automatically splits by `<FILL_ME>`](https://huggingface.co/docs/transformers/main/model_doc/code_llama#transformers.CodeLlamaTokenizer.fill_token) to create a formatted input string that follows [the original training pattern](https://github.com/facebookresearch/codellama/blob/cb51c14ec761370ba2e2bc351374a79265d0465e/llama/generation.py#L402). This is more robust than preparing the pattern yourself: it avoids pitfalls, such as token glueing, that are very hard to debug.  To see how much CPU and GPU memory you need for this model or others, try [this calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) which can help determine that value.

The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string.

<Tip>

Code Llama has the same architecture as the `Llama2` models, refer to [Llama2's documentation page](llama2) for the API reference.
Find Code Llama tokenizer reference below. 
</Tip>


## CodeLlamaTokenizer

[[autodoc]] CodeLlamaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## CodeLlamaTokenizerFast

[[autodoc]] CodeLlamaTokenizerFast
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - update_post_processor
    - save_vocabulary


key: _node_content
string_value:
{"id_": "91ef34b8-de71-4e63-8ca1-3efb20e120c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3d0e32f4-8d79-4b00-a03e-be6deb08eb05", "node_type": "1", "metadata": {}, "hash": "cc5ae288ea2785bdc6fb7d714bab44b9ea38ed6094d3600f61067eb252957c31", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3173, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLaVA-NeXT

## Overview

The LLaVA-NeXT model was proposed in [LLaVA-NeXT: Improved reasoning, OCR, and world knowledge](https://llava-vl.github.io/blog/2024-01-30-llava-next/) by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon [LLaVa](llava) by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning.

The introduction from the blog is the following:

*In October 2023, we released LLaVA-1.5 with a simple and efficient design along with great performance on a benchmark suite of 12 datasets. It has since served as the foundation of many comprehensive studies of data, model, and capabilities of large multimodal models (LMM), and has enabled various new applications.

Today, we are thrilled to present LLaVA-NeXT, with improved reasoning, OCR, and world knowledge. LLaVA-NeXT even exceeds Gemini Pro on several benchmarks.

Compared with LLaVA-1.5, LLaVA-NeXT has several improvements:

Increasing the input image resolution to 4x more pixels. This allows it to grasp more visual details. It supports three aspect ratios, up to 672x672, 336x1344, 1344x336 resolution.
Better visual reasoning and OCR capability with an improved visual instruction tuning data mixture.
Better visual conversation for more scenarios, covering different applications. Better world knowledge and logical reasoning.
Efficient deployment and inference with SGLang.
Along with performance improvements, LLaVA-NeXT maintains the minimalist design and data efficiency of LLaVA-1.5. It re-uses the pretrained connector of LLaVA-1.5, and still uses less than 1M visual instruction tuning samples. The largest 34B variant finishes training in ~1 day with 32 A100s.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/llava_next_overview.png"
alt="drawing" width="600"/>

<small> LLaVa-NeXT incorporates a higher input resolution by encoding various patches of the input image. Taken from the <a href="https://arxiv.org/abs/2310.03744">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/haotian-liu/LLaVA/tree/main).

## Usage tips

- We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating.

<Tip warning={true}>

- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is "left-padding" if model is in `eval()` mode, otherwise "right-padding".

</Tip>


- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt.


key: _node_content
string_value:
{"id_": "3d0e32f4-8d79-4b00-a03e-be6deb08eb05", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "91ef34b8-de71-4e63-8ca1-3efb20e120c1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "c282bd1e257146a4279ba27ebb40eaf5ad2cdcb92aef98525c3b1c6e3a6f103b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "89bb0620-e28b-4c22-92e6-abff7fc57902", "node_type": "1", "metadata": {}, "hash": "6bbc7058afad91efee48d0ea76ea00afef82cac144a896939aee01c1be48e142", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2851, "end_char_idx": 5322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use the processor's `apply_chat_template` to format your prompts correctly. For that you have to construct a conversation history, passing a plain string will not format your prompt. Each message in the conversation history for chat templates is a dictionary with keys "role" and "content". The "content" should be a list of dictionaries, for "text" and "image" modalities. Below is an example of how to do that and the list of formats accepted by each checkpoint.

We will use [llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) and a conversation history of text and image. Each content field has to be a list of dicts, as follows:

```python
from transformers import LlavaNextProcessor

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What’s shown in this image?"},
        ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image in more details."},
        ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
>>> "[INST] <image>\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]"
```

- If you want to construct a chat prompt yourself, below is a list of possible formats
.
[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:
```bash
"[INST] <image>\nWhat is shown in this image? [/INST]"
```

[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:
```bash
"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image?


key: _node_content
string_value:
{"id_": "89bb0620-e28b-4c22-92e6-abff7fc57902", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3d0e32f4-8d79-4b00-a03e-be6deb08eb05", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "254ea0b82db6462380d7a8b45223e081eca9fb2941c0feacbf2969c20c889db4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0625fcdf-fa51-4d39-8048-d99850beacbf", "node_type": "1", "metadata": {}, "hash": "2190ce1a5a0580b4142e1f1700041ba9209f48bc717868207d13d73e46269dd7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3963, "end_char_idx": 5808, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
        ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image in more details."},
        ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
>>> "[INST] <image>\nWhat's shown in this image? [/INST] This image shows a red stop sign. [INST] Describe the image in more details. [/INST]"
```

- If you want to construct a chat prompt yourself, below is a list of possible formats
.
[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:
```bash
"[INST] <image>\nWhat is shown in this image? [/INST]"
```

[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:
```bash
"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"
```

[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:
```bash
"<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|><|im_start|>assistant\n"
```

[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:

```bash
"<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful language and vision assistant.


key: _node_content
string_value:
{"id_": "0625fcdf-fa51-4d39-8048-d99850beacbf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89bb0620-e28b-4c22-92e6-abff7fc57902", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "6696e29c53ed9a5fab95b30e368d05bcfe827191ab05a0e43a1ac0c234b8005f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9eb4208a-0f98-423f-98f4-5295448ed47e", "node_type": "1", "metadata": {}, "hash": "224153815aaa13977fed0a4b0237e8192d02fe2b7a912db04b49382f8c8cfbfe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4727, "end_char_idx": 7415, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[llava-v1.6-mistral-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) requires the following format:
```bash
"[INST] <image>\nWhat is shown in this image? [/INST]"
```

[llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) and [llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) require the following format:
```bash
"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"
```

[llava-v1.6-34b-hf](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) requires the following format:
```bash
"<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|><|im_start|>assistant\n"
```

[llama3-llava-next-8b-hf](https://huggingface.co/llava-hf/llava-next-8b-hf) requires the following format:

```bash
"<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.<|eot_id|><|start_header_id|><|start_header_id|>user<|end_header_id|>\n\n<image>\nWhat is shown in this image?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
```

[llava-next-72b-hf](https://huggingface.co/llava-hf/llava-next-72b-hf) and [llava-next-110b-hf](https://huggingface.co/llava-hf/llava-next-110b-hf) require the following format:

```bash
"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|>\n<|im_start|>assistant\n"
```

## Usage example

### Single image inference

Here's how to load the model and perform inference in half-precision (`torch.float16`):

```python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import requests

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) 
model.to("cuda:0")

# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?


key: _node_content
string_value:
{"id_": "9eb4208a-0f98-423f-98f4-5295448ed47e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0625fcdf-fa51-4d39-8048-d99850beacbf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "f26031ae50fe8b3764819ba9b0deb5c034e4bfe6c81e7ccbbe41fb55c2ba2769", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2869a39-935c-4b1b-8f1b-5273ae25200c", "node_type": "1", "metadata": {}, "hash": "be4ca95a0b44117d08a0aa9842f2c7e3c6ac292766840432bc7182507c2c2309", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7415, "end_char_idx": 10499, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)

print(processor.decode(output[0], skip_special_tokens=True))
```

### Multi image inference

LLaVa-Next can perform inference with multiple images as input, where images either belong to the same prompt or different prompts (in batched inference). Here is how you can do it:

```python
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LlavaNextForConditionalGeneration

# Load the model in half-precision
model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")

# Get three different images
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image_stop = Image.open(requests.get(url, stream=True).raw)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_cats = Image.open(requests.get(url, stream=True).raw)

url = "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"
image_snowman = Image.open(requests.get(url, stream=True).raw)

# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is not
conversation_1 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"},
            ],
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "There is a red stop sign in the image."},
            ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What about this image? How many cats do you see?"},
            ],
    },
]

conversation_2 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"},
            ],
    },
]

prompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)
prompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)
prompts = [prompt_1, prompt_2]

# We can simply feed images in the order they have to be used in the text prompt
# Each "<image>" token uses one image leaving the next for the subsequent "<image>" tokens
inputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors="pt").to(model.device)

# Generate
generate_ids = model.generate(**inputs, max_new_tokens=30)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
```

## Model optimization

### Quantization using Bitsandbytes

The model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model.


key: _node_content
string_value:
{"id_": "d2869a39-935c-4b1b-8f1b-5273ae25200c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9990cdb1-3b28-4fad-aa10-e0ddc6620d23", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "08f02de732c0fc26ec8f972528e3fc9ac13140f50fbf752aaad6c0c5bc4c2150", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9eb4208a-0f98-423f-98f4-5295448ed47e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next", "title": "LLaVA-NeXT", "tokens": 3301, "retrieve_doc": true, "source": "transformers"}, "hash": "c4f43b74cb54062f9412bae37a6ef34259a7cb1912cf12e82745f8a203de9447", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9043, "end_char_idx": 11900, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
            ],
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "There is a red stop sign in the image."},
            ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What about this image? How many cats do you see?"},
            ],
    },
]

conversation_2 = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What is shown in this image?"},
            ],
    },
]

prompt_1 = processor.apply_chat_template(conversation_1, add_generation_prompt=True)
prompt_2 = processor.apply_chat_template(conversation_2, add_generation_prompt=True)
prompts = [prompt_1, prompt_2]

# We can simply feed images in the order they have to be used in the text prompt
# Each "<image>" token uses one image leaving the next for the subsequent "<image>" tokens
inputs = processor(text=prompts, images=[image_stop, image_cats, image_snowman], padding=True, return_tensors="pt").to(model.device)

# Generate
generate_ids = model.generate(**inputs, max_new_tokens=30)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
```

## Model optimization

### Quantization using Bitsandbytes

The model can be loaded in 8 or 4 bits, greatly reducing the memory requirements while maintaining the performance of the original model. First make sure to install bitsandbytes, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:

```python
from transformers import LlavaNextForConditionalGeneration, BitsAndBytesConfig

# specify how to quantize the model
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", quantization_config=quantization_config, device_map="auto")
```

### Use Flash-Attention 2 to further speed-up generation

First make sure to install flash-attn. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:

```python
from transformers import LlavaNextForConditionalGeneration

model = LlavaNextForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True,
    use_flash_attention_2=True
).to(0)
```

## LlavaNextConfig

[[autodoc]] LlavaNextConfig

## LlavaNextImageProcessor

[[autodoc]] LlavaNextImageProcessor
    - preprocess

## LlavaNextProcessor

[[autodoc]] LlavaNextProcessor

## LlavaNextForConditionalGeneration

[[autodoc]] LlavaNextForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "a66ca2d4-5805-445e-9bb1-be5fe3d208b3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers"}, "hash": "1fafa44d10a88b34cb80261b380c21dac121df2bc2cb3a95fcb70e0b5f7bc99a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dad91187-8fa2-47ec-80b7-2d0f775ebdc7", "node_type": "1", "metadata": {}, "hash": "cb48e1f95e492b18be7b03a8e5287b16e9de60040679cebea38817c967aa47a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2118, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XGLM

## Overview

The XGLM model was proposed in [Few-shot Learning with Multilingual Language Models](https://arxiv.org/abs/2112.10668)
by Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, 
Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, 
Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li.

The abstract from the paper is the following:

*Large-scale autoregressive language models such as GPT-3 are few-shot learners that can perform a wide range of language 
tasks without fine-tuning. While these models are known to be able to jointly represent many different languages, 
their training data is dominated by English, potentially limiting their cross-lingual generalization. 
In this work, we train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages, 
and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters 
sets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size 
in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) 
and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, 
our model outperforms GPT-3 on 171 out of 182 translation directions with 32 training examples, while surpassing the 
official supervised baseline in 45 directions. We present a detailed analysis of where the model succeeds and fails, 
showing in particular that it enables cross-lingual in-context learning on some tasks, while there is still room for improvement 
on surface form robustness and adaptation to tasks that do not have a natural cloze form. Finally, we evaluate our models 
in social value tasks such as hate speech detection in five languages and find it has limitations similar to comparable sized GPT-3 models.*


key: _node_content
string_value:
{"id_": "dad91187-8fa2-47ec-80b7-2d0f775ebdc7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f13959af-548a-463d-bc41-1e99ebf7d10a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers"}, "hash": "1fafa44d10a88b34cb80261b380c21dac121df2bc2cb3a95fcb70e0b5f7bc99a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a66ca2d4-5805-445e-9bb1-be5fe3d208b3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xglm", "title": "XGLM", "tokens": 999, "retrieve_doc": true, "source": "transformers"}, "hash": "51739c7628dc650501961767272cb7ab9044f6c65e6195b14ec6b73ed81f06db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2121, "end_char_idx": 3085, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [Suraj](https://huggingface.co/valhalla). The original code can be found [here](https://github.com/pytorch/fairseq/tree/main/examples/xglm).

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## XGLMConfig

[[autodoc]] XGLMConfig

## XGLMTokenizer

[[autodoc]] XGLMTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## XGLMTokenizerFast

[[autodoc]] XGLMTokenizerFast

<frameworkcontent>
<pt>

## XGLMModel

[[autodoc]] XGLMModel
    - forward

## XGLMForCausalLM

[[autodoc]] XGLMForCausalLM
    - forward

</pt>
<tf>

## TFXGLMModel

[[autodoc]] TFXGLMModel
    - call

## TFXGLMForCausalLM

[[autodoc]] TFXGLMForCausalLM
    - call

</tf>
<jax>

## FlaxXGLMModel

[[autodoc]] FlaxXGLMModel
    - __call__

## FlaxXGLMForCausalLM

[[autodoc]] FlaxXGLMForCausalLM
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "3215fb88-c1fa-4b61-8670-42817324616f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f119c223-3dfd-4f5c-b63c-c2a5eef0ad99", "node_type": "1", "metadata": {}, "hash": "24ba341ce000d143f08391e307bb8f97cdc356cf1567f293166fd7b31df73d7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2885, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LayoutLMV2

## Overview

The LayoutLMV2 model was proposed in [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://arxiv.org/abs/2012.14740) by Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu,
Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou. LayoutLMV2 improves [LayoutLM](layoutlm) to obtain
state-of-the-art results across several document image understanding benchmarks:

- information extraction from scanned documents: the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (a
  collection of 199 annotated forms comprising more than 30,000 words), the [CORD](https://github.com/clovaai/cord)
  dataset (a collection of 800 receipts for training, 100 for validation and 100 for testing), the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset (a collection of 626 receipts for training and 347 receipts for testing)
  and the [Kleister-NDA](https://github.com/applicaai/kleister-nda) dataset (a collection of non-disclosure
  agreements from the EDGAR database, including 254 documents for training, 83 documents for validation, and 203
  documents for testing).
- document image classification: the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset (a collection of
  400,000 images belonging to one of 16 classes).
- document visual question answering: the [DocVQA](https://arxiv.org/abs/2007.00398) dataset (a collection of 50,000
  questions defined on 12,000+ document images).

The abstract from the paper is the following:

*Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to
its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. In this
paper, we present LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model
architectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked
visual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training
stage, where cross-modality interaction is better learned. Meanwhile, it also integrates a spatial-aware self-attention
mechanism into the Transformer architecture, so that the model can fully understand the relative positional
relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms strong baselines and
achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks,
including FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852),
RVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). The pre-trained LayoutLMv2 model is publicly available at
this https URL.*

LayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`.


key: _node_content
string_value:
{"id_": "f119c223-3dfd-4f5c-b63c-c2a5eef0ad99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3215fb88-c1fa-4b61-8670-42817324616f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "986397e737f6cc88422c2e044e9593faa1bb6d88f378ffb6765e9fa4dcb14f12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "55c1bee6-bce8-40c5-8dd5-b1f75ce60363", "node_type": "1", "metadata": {}, "hash": "451899999b7387093a3b39221144539623e1f6e72ead6d29bf01c3b0a6686131", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1317, "end_char_idx": 4294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- document visual question answering: the [DocVQA](https://arxiv.org/abs/2007.00398) dataset (a collection of 50,000
  questions defined on 12,000+ document images).

The abstract from the paper is the following:

*Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to
its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. In this
paper, we present LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model
architectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked
visual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training
stage, where cross-modality interaction is better learned. Meanwhile, it also integrates a spatial-aware self-attention
mechanism into the Transformer architecture, so that the model can fully understand the relative positional
relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms strong baselines and
achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks,
including FUNSD (0.7895 -> 0.8420), CORD (0.9493 -> 0.9601), SROIE (0.9524 -> 0.9781), Kleister-NDA (0.834 -> 0.852),
RVL-CDIP (0.9443 -> 0.9564), and DocVQA (0.7295 -> 0.8672). The pre-trained LayoutLMv2 model is publicly available at
this https URL.*

LayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the
following to install them:
```bash
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -m pip install torchvision tesseract
```
(If you are developing for LayoutLMv2, note that passing the doctests also requires the installation of these packages.)

## Usage tips

- The main difference between LayoutLMv1 and LayoutLMv2 is that the latter incorporates visual embeddings during
  pre-training (while LayoutLMv1 only adds visual embeddings during fine-tuning).
- LayoutLMv2 adds both a relative 1D attention bias as well as a spatial 2D attention bias to the attention scores in
  the self-attention layers. Details can be found on page 5 of the [paper](https://arxiv.org/abs/2012.14740).
- Demo notebooks on how to use the LayoutLMv2 model on RVL-CDIP, FUNSD, DocVQA, CORD can be found [here](https://github.com/NielsRogge/Transformers-Tutorials).
- LayoutLMv2 uses Facebook AI's [Detectron2](https://github.com/facebookresearch/detectron2/) package for its visual
  backbone. See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation
  instructions.
- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely
  `image` and `bbox`. The `image` input corresponds to the original document image in which the text
  tokens occur. The model expects each document image to be of size 224x224.


key: _node_content
string_value:
{"id_": "55c1bee6-bce8-40c5-8dd5-b1f75ce60363", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f119c223-3dfd-4f5c-b63c-c2a5eef0ad99", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "0e7b0c6a181e27aad01dcfb8db6cff9dc033898c2672c02024a421c15c02a834", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9c5cbd70-eb19-4513-b02e-d0db42fc357c", "node_type": "1", "metadata": {}, "hash": "c6f8af7ec45894bf1b5fc87695831eea4dd059bf39fbfe266086db47b0d214cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2743, "end_char_idx": 5351, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The pre-trained LayoutLMv2 model is publicly available at
this https URL.*

LayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the
following to install them:
```bash
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -m pip install torchvision tesseract
```
(If you are developing for LayoutLMv2, note that passing the doctests also requires the installation of these packages.)

## Usage tips

- The main difference between LayoutLMv1 and LayoutLMv2 is that the latter incorporates visual embeddings during
  pre-training (while LayoutLMv1 only adds visual embeddings during fine-tuning).
- LayoutLMv2 adds both a relative 1D attention bias as well as a spatial 2D attention bias to the attention scores in
  the self-attention layers. Details can be found on page 5 of the [paper](https://arxiv.org/abs/2012.14740).
- Demo notebooks on how to use the LayoutLMv2 model on RVL-CDIP, FUNSD, DocVQA, CORD can be found [here](https://github.com/NielsRogge/Transformers-Tutorials).
- LayoutLMv2 uses Facebook AI's [Detectron2](https://github.com/facebookresearch/detectron2/) package for its visual
  backbone. See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation
  instructions.
- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely
  `image` and `bbox`. The `image` input corresponds to the original document image in which the text
  tokens occur. The model expects each document image to be of size 224x224. This means that if you have a batch of
  document images, `image` should be a tensor of shape (batch_size, 3, 224, 224). This can be either a
  `torch.Tensor` or a `Detectron2.structures.ImageList`. You don't need to normalize the channels, as this is
  done by the model. Important to note is that the visual backbone expects BGR channels instead of RGB, as all models
  in Detectron2 are pre-trained using the BGR format. The `bbox` input are the bounding boxes (i.e. 2D-positions)
  of the input text tokens. This is identical to [`LayoutLMModel`]. These can be obtained using an
  external OCR engine such as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python
  wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1)
  format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1)
  represents the position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on
  a 0-1000 scale.


key: _node_content
string_value:
{"id_": "9c5cbd70-eb19-4513-b02e-d0db42fc357c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55c1bee6-bce8-40c5-8dd5-b1f75ce60363", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "5a3b68a93396a806c340a6ab136f3a5bfbb4637d50028d658044368dadebfe19", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4c432a9-8c7f-4215-a862-2bd250b324d5", "node_type": "1", "metadata": {}, "hash": "d531aa9498bf8ffc564af98eb5777ea26e6e269f5bf33527401c728f1a993fc6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3905, "end_char_idx": 6806, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See [this link](https://detectron2.readthedocs.io/en/latest/tutorials/install.html) for installation
  instructions.
- In addition to `input_ids`, [`~LayoutLMv2Model.forward`] expects 2 additional inputs, namely
  `image` and `bbox`. The `image` input corresponds to the original document image in which the text
  tokens occur. The model expects each document image to be of size 224x224. This means that if you have a batch of
  document images, `image` should be a tensor of shape (batch_size, 3, 224, 224). This can be either a
  `torch.Tensor` or a `Detectron2.structures.ImageList`. You don't need to normalize the channels, as this is
  done by the model. Important to note is that the visual backbone expects BGR channels instead of RGB, as all models
  in Detectron2 are pre-trained using the BGR format. The `bbox` input are the bounding boxes (i.e. 2D-positions)
  of the input text tokens. This is identical to [`LayoutLMModel`]. These can be obtained using an
  external OCR engine such as Google's [Tesseract](https://github.com/tesseract-ocr/tesseract) (there's a [Python
  wrapper](https://pypi.org/project/pytesseract/) available). Each bounding box should be in (x0, y0, x1, y1)
  format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1)
  represents the position of the lower right corner. Note that one first needs to normalize the bounding boxes to be on
  a 0-1000 scale. To normalize, you can use the following function:

```python
def normalize_bbox(bbox, width, height):
    return [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
```

Here, `width` and `height` correspond to the width and height of the original document in which the token
occurs (before resizing the image). Those can be obtained using the Python Image Library (PIL) library for example, as
follows:

```python
from PIL import Image

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
)

width, height = image.size
```

However, this model includes a brand new [`~transformers.LayoutLMv2Processor`] which can be used to directly
prepare data for the model (including applying OCR under the hood). More information can be found in the "Usage"
section below.

- Internally, [`~transformers.LayoutLMv2Model`] will send the `image` input through its visual backbone to
  obtain a lower-resolution feature map, whose shape is equal to the `image_feature_pool_shape` attribute of
  [`~transformers.LayoutLMv2Config`]. This feature map is then flattened to obtain a sequence of image tokens. As
  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text
  tokens, and send through the Transformer encoder.


key: _node_content
string_value:
{"id_": "c4c432a9-8c7f-4215-a862-2bd250b324d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9c5cbd70-eb19-4513-b02e-d0db42fc357c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "caf80668ee15fa5716183094280addbfd8a45e1020911b811c3c7745bc6e4a0a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4babf1e6-475e-4e25-978f-e2dcde4c021c", "node_type": "1", "metadata": {}, "hash": "4ac22479df246c933397775b5a5d15f9422f5e3339f8c9e576915face30fbae1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5267, "end_char_idx": 8127, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that one first needs to normalize the bounding boxes to be on
  a 0-1000 scale. To normalize, you can use the following function:

```python
def normalize_bbox(bbox, width, height):
    return [
        int(1000 * (bbox[0] / width)),
        int(1000 * (bbox[1] / height)),
        int(1000 * (bbox[2] / width)),
        int(1000 * (bbox[3] / height)),
    ]
```

Here, `width` and `height` correspond to the width and height of the original document in which the token
occurs (before resizing the image). Those can be obtained using the Python Image Library (PIL) library for example, as
follows:

```python
from PIL import Image

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
)

width, height = image.size
```

However, this model includes a brand new [`~transformers.LayoutLMv2Processor`] which can be used to directly
prepare data for the model (including applying OCR under the hood). More information can be found in the "Usage"
section below.

- Internally, [`~transformers.LayoutLMv2Model`] will send the `image` input through its visual backbone to
  obtain a lower-resolution feature map, whose shape is equal to the `image_feature_pool_shape` attribute of
  [`~transformers.LayoutLMv2Config`]. This feature map is then flattened to obtain a sequence of image tokens. As
  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text
  tokens, and send through the Transformer encoder. This means that the last hidden states of the model will have a
  length of 512 + 49 = 561, if you pad the text tokens up to the max length. More generally, the last hidden states
  will have a shape of `seq_length` + `image_feature_pool_shape[0]` *
  `config.image_feature_pool_shape[1]`.
- When calling [`~transformers.LayoutLMv2Model.from_pretrained`], a warning will be printed with a long list of
  parameter names that are not initialized. This is not a problem, as these parameters are batch normalization
  statistics, which are going to have values when fine-tuning on a custom dataset.
- If you want to train the model in a distributed environment, make sure to call [`synchronize_batch_norm`] on the
  model in order to properly synchronize the batch normalization layers of the visual backbone.

In addition, there's LayoutXLM, which is a multilingual version of LayoutLMv2. More information can be found on
[LayoutXLM's documentation page](layoutxlm).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLMv2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "4babf1e6-475e-4e25-978f-e2dcde4c021c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4c432a9-8c7f-4215-a862-2bd250b324d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "322fe81d5489bad6bd4e4f4f7f80622e561f49af479448f4454549e7fff133b7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bbcd785a-3965-4075-98ea-4175b1499a7c", "node_type": "1", "metadata": {}, "hash": "71bc6e21020fea58f988daed6ca08706339e1d1e6ca9b2a9c52a0100c8cae53e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6559, "end_char_idx": 9560, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This feature map is then flattened to obtain a sequence of image tokens. As
  the size of the feature map is 7x7 by default, one obtains 49 image tokens. These are then concatenated with the text
  tokens, and send through the Transformer encoder. This means that the last hidden states of the model will have a
  length of 512 + 49 = 561, if you pad the text tokens up to the max length. More generally, the last hidden states
  will have a shape of `seq_length` + `image_feature_pool_shape[0]` *
  `config.image_feature_pool_shape[1]`.
- When calling [`~transformers.LayoutLMv2Model.from_pretrained`], a warning will be printed with a long list of
  parameter names that are not initialized. This is not a problem, as these parameters are batch normalization
  statistics, which are going to have values when fine-tuning on a custom dataset.
- If you want to train the model in a distributed environment, make sure to call [`synchronize_batch_norm`] on the
  model in order to properly synchronize the batch normalization layers of the visual backbone.

In addition, there's LayoutXLM, which is a multilingual version of LayoutLMv2. More information can be found on
[LayoutXLM's documentation page](layoutxlm).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLMv2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A notebook on how to [finetune LayoutLMv2 for text-classification on RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).
- See also: [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="question-answering"/>

- A notebook on how to [finetune LayoutLMv2 for question-answering on DocVQA dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).
- See also: [Question answering task guide](../tasks/question_answering)
- See also: [Document question answering task guide](../tasks/document_question_answering)


<PipelineTag pipeline="token-classification"/>

- A notebook on how to [finetune LayoutLMv2 for token-classification on CORD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/CORD/Fine_tuning_LayoutLMv2ForTokenClassification_on_CORD.ipynb).
- A notebook on how to [finetune LayoutLMv2 for token-classification on FUNSD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb).


key: _node_content
string_value:
{"id_": "bbcd785a-3965-4075-98ea-4175b1499a7c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4babf1e6-475e-4e25-978f-e2dcde4c021c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "fef0c9a3a87404af277ea75bf4a153d3e8b0944d4d7aea5231f1b172118d2515", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f6e964a-a9e6-4902-9362-c9ad9b19f9dc", "node_type": "1", "metadata": {}, "hash": "47215afe5810e740365e4e1b698be9b7c2cfee049a0a5bbad17937c92b0c6300", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8029, "end_char_idx": 11098, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A notebook on how to [finetune LayoutLMv2 for text-classification on RVL-CDIP dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).
- See also: [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="question-answering"/>

- A notebook on how to [finetune LayoutLMv2 for question-answering on DocVQA dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).
- See also: [Question answering task guide](../tasks/question_answering)
- See also: [Document question answering task guide](../tasks/document_question_answering)


<PipelineTag pipeline="token-classification"/>

- A notebook on how to [finetune LayoutLMv2 for token-classification on CORD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/CORD/Fine_tuning_LayoutLMv2ForTokenClassification_on_CORD.ipynb).
- A notebook on how to [finetune LayoutLMv2 for token-classification on FUNSD dataset](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb).
- See also: [Token classification task guide](../tasks/token_classification)

## Usage: LayoutLMv2Processor

The easiest way to prepare data for the model is to use [`LayoutLMv2Processor`], which internally
combines a image processor ([`LayoutLMv2ImageProcessor`]) and a tokenizer
([`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]). The image processor
handles the image modality, while the tokenizer handles the text modality. A processor combines both, which is ideal
for a multi-modal model like LayoutLMv2. Note that you can still use both separately, if you only want to handle one
modality.

```python
from transformers import LayoutLMv2ImageProcessor, LayoutLMv2TokenizerFast, LayoutLMv2Processor

image_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default
tokenizer = LayoutLMv2TokenizerFast.from_pretrained("microsoft/layoutlmv2-base-uncased")
processor = LayoutLMv2Processor(image_processor, tokenizer)
```

In short, one can provide a document image (and possibly additional data) to [`LayoutLMv2Processor`],
and it will create the inputs expected by the model. Internally, the processor first uses
[`LayoutLMv2ImageProcessor`] to apply OCR on the image to get a list of words and normalized
bounding boxes, as well to resize the image to a given size in order to get the `image` input. The words and
normalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or
[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,
`attention_mask`, `token_type_ids`, `bbox`.


key: _node_content
string_value:
{"id_": "5f6e964a-a9e6-4902-9362-c9ad9b19f9dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bbcd785a-3965-4075-98ea-4175b1499a7c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "a8336422e5e4a2cf5763c5fb7d40f4b1b8a9b4e87f71d84addf6d6c5240abe2b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dfe0a283-7582-4ac8-9451-0758aae7fd3c", "node_type": "1", "metadata": {}, "hash": "83d4b2cb5c4e4058b228cfa676b68d244f8ced8c4f34f64052425d42a0381d5b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9561, "end_char_idx": 12350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- See also: [Token classification task guide](../tasks/token_classification)

## Usage: LayoutLMv2Processor

The easiest way to prepare data for the model is to use [`LayoutLMv2Processor`], which internally
combines a image processor ([`LayoutLMv2ImageProcessor`]) and a tokenizer
([`LayoutLMv2Tokenizer`] or [`LayoutLMv2TokenizerFast`]). The image processor
handles the image modality, while the tokenizer handles the text modality. A processor combines both, which is ideal
for a multi-modal model like LayoutLMv2. Note that you can still use both separately, if you only want to handle one
modality.

```python
from transformers import LayoutLMv2ImageProcessor, LayoutLMv2TokenizerFast, LayoutLMv2Processor

image_processor = LayoutLMv2ImageProcessor()  # apply_ocr is set to True by default
tokenizer = LayoutLMv2TokenizerFast.from_pretrained("microsoft/layoutlmv2-base-uncased")
processor = LayoutLMv2Processor(image_processor, tokenizer)
```

In short, one can provide a document image (and possibly additional data) to [`LayoutLMv2Processor`],
and it will create the inputs expected by the model. Internally, the processor first uses
[`LayoutLMv2ImageProcessor`] to apply OCR on the image to get a list of words and normalized
bounding boxes, as well to resize the image to a given size in order to get the `image` input. The words and
normalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or
[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,
`attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide word labels to the processor,
which are turned into token-level `labels`.

[`LayoutLMv2Processor`] uses [PyTesseract](https://pypi.org/project/pytesseract/), a Python
wrapper around Google's Tesseract OCR engine, under the hood. Note that you can still use your own OCR engine of
choice, and provide the words and normalized boxes yourself. This requires initializing
[`LayoutLMv2ImageProcessor`] with `apply_ocr` set to `False`.

In total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these
use cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).

**Use case 1: document image classification (training, inference) + token classification (inference), apply_ocr =
True**

This is the simplest case, in which the processor (actually the image processor) will perform OCR on the image to get
the words and normalized bounding boxes.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."


key: _node_content
string_value:
{"id_": "dfe0a283-7582-4ac8-9451-0758aae7fd3c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f6e964a-a9e6-4902-9362-c9ad9b19f9dc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "b789baa80406ed679ad91e913a574e1b14661598ecedbcfe0f19bf85de6cd4c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4db6da7-b3e2-40ec-a84b-144b178c764b", "node_type": "1", "metadata": {}, "hash": "9cc199a8fd3a99d1b946c4e20a2cae2f1bbe18e070ec31e62e4470b77e5eff0e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10890, "end_char_idx": 14026, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The words and
normalized bounding boxes are then provided to [`LayoutLMv2Tokenizer`] or
[`LayoutLMv2TokenizerFast`], which converts them to token-level `input_ids`,
`attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide word labels to the processor,
which are turned into token-level `labels`.

[`LayoutLMv2Processor`] uses [PyTesseract](https://pypi.org/project/pytesseract/), a Python
wrapper around Google's Tesseract OCR engine, under the hood. Note that you can still use your own OCR engine of
choice, and provide the words and normalized boxes yourself. This requires initializing
[`LayoutLMv2ImageProcessor`] with `apply_ocr` set to `False`.

In total, there are 5 use cases that are supported by the processor. Below, we list them all. Note that each of these
use cases work for both batched and non-batched inputs (we illustrate them for non-batched inputs).

**Use case 1: document image classification (training, inference) + token classification (inference), apply_ocr =
True**

This is the simplest case, in which the processor (actually the image processor) will perform OCR on the image to get
the words and normalized bounding boxes.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
encoding = processor(
    image, return_tensors="pt"
)  # you can also add all tokenizer parameters here such as padding, truncation
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

**Use case 2: document image classification (training, inference) + token classification (inference), apply_ocr=False**

In case one wants to do OCR themselves, one can initialize the image processor with `apply_ocr` set to
`False`. In that case, one should provide the words and corresponding (normalized) bounding boxes themselves to
the processor.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
encoding = processor(image, words, boxes=boxes, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

**Use case 3: token classification (training), apply_ocr=False**

For token classification tasks (such as FUNSD, CORD, SROIE, Kleister-NDA), one can also provide the corresponding word
labels in order to train a model. The processor will then convert these into token-level `labels`. By default, it
will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the
`ignore_index` of PyTorch's CrossEntropyLoss.


key: _node_content
string_value:
{"id_": "e4db6da7-b3e2-40ec-a84b-144b178c764b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dfe0a283-7582-4ac8-9451-0758aae7fd3c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "bbc052b043393f9c8d1fc74795cdf6be4921202006b19c7635b7b38e953d7dec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "966d5fc8-2a46-436f-9ba5-550715dd1666", "node_type": "1", "metadata": {}, "hash": "bffa87630189c671fd705e6073f4ea42c511f3deb80a825f0e5982ffae7547a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12842, "end_char_idx": 15797, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In that case, one should provide the words and corresponding (normalized) bounding boxes themselves to
the processor.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
encoding = processor(image, words, boxes=boxes, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

**Use case 3: token classification (training), apply_ocr=False**

For token classification tasks (such as FUNSD, CORD, SROIE, Kleister-NDA), one can also provide the corresponding word
labels in order to train a model. The processor will then convert these into token-level `labels`. By default, it
will only label the first wordpiece of a word, and label the remaining wordpieces with -100, which is the
`ignore_index` of PyTorch's CrossEntropyLoss. In case you want all wordpieces of a word to be labeled, you can
initialize the tokenizer with `only_label_first_subword` set to `False`.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
word_labels = [1, 2]
encoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])
```

**Use case 4: visual question answering (inference), apply_ocr=True**

For visual question answering tasks (such as DocVQA), you can provide a question to the processor. By default, the
processor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
question = "What's his name?"
encoding = processor(image, question, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

**Use case 5: visual question answering (inference), apply_ocr=False**

For visual question answering tasks (such as DocVQA), you can provide a question to the processor.


key: _node_content
string_value:
{"id_": "966d5fc8-2a46-436f-9ba5-550715dd1666", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4db6da7-b3e2-40ec-a84b-144b178c764b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "ab6e96adc85940729724dacff944092812a585b1b972646ac100aa60637b12f0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30ec7155-1a43-45a2-8b38-378ed6c848a6", "node_type": "1", "metadata": {}, "hash": "cbc8471dd3a1451e287a7687a0ea2e0e1546e1f9d982554a815907b38e30e044", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14422, "end_char_idx": 16274, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
of your documents (PDFs must be converted to images)."
).convert("RGB")
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
word_labels = [1, 2]
encoding = processor(image, words, boxes=boxes, word_labels=word_labels, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'labels', 'image'])
```

**Use case 4: visual question answering (inference), apply_ocr=True**

For visual question answering tasks (such as DocVQA), you can provide a question to the processor. By default, the
processor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
question = "What's his name?"
encoding = processor(image, question, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

**Use case 5: visual question answering (inference), apply_ocr=False**

For visual question answering tasks (such as DocVQA), you can provide a question to the processor. If you want to
perform OCR yourself, you can provide your own words and (normalized) bounding boxes to the processor.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
question = "What's his name?"


key: _node_content
string_value:
{"id_": "30ec7155-1a43-45a2-8b38-378ed6c848a6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c3999cb4-27da-4160-9f57-8e2fbb75c610", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "7f6e83f2f7a8b7b3139399df24e3e772b1388211eb086a489e4a599a117b6c28", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "966d5fc8-2a46-436f-9ba5-550715dd1666", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv2", "title": "LayoutLMV2", "tokens": 4571, "retrieve_doc": true, "source": "transformers"}, "hash": "f06ac39622f1608d01d0986dde382db55af518362654ba06f4e661af7e9e8bcd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15007, "end_char_idx": 17351, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
By default, the
processor will apply OCR on the image, and create [CLS] question tokens [SEP] word tokens [SEP].

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
question = "What's his name?"
encoding = processor(image, question, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

**Use case 5: visual question answering (inference), apply_ocr=False**

For visual question answering tasks (such as DocVQA), you can provide a question to the processor. If you want to
perform OCR yourself, you can provide your own words and (normalized) bounding boxes to the processor.

```python
from transformers import LayoutLMv2Processor
from PIL import Image

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

image = Image.open(
    "name_of_your_document - can be a png, jpg, etc. of your documents (PDFs must be converted to images)."
).convert("RGB")
question = "What's his name?"
words = ["hello", "world"]
boxes = [[1, 2, 3, 4], [5, 6, 7, 8]]  # make sure to normalize your bounding boxes
encoding = processor(image, question, words, boxes=boxes, return_tensors="pt")
print(encoding.keys())
# dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'bbox', 'image'])
```

## LayoutLMv2Config

[[autodoc]] LayoutLMv2Config

## LayoutLMv2FeatureExtractor

[[autodoc]] LayoutLMv2FeatureExtractor
    - __call__

## LayoutLMv2ImageProcessor

[[autodoc]] LayoutLMv2ImageProcessor
    - preprocess

## LayoutLMv2Tokenizer

[[autodoc]] LayoutLMv2Tokenizer
    - __call__
    - save_vocabulary

## LayoutLMv2TokenizerFast

[[autodoc]] LayoutLMv2TokenizerFast
    - __call__

## LayoutLMv2Processor

[[autodoc]] LayoutLMv2Processor
    - __call__

## LayoutLMv2Model

[[autodoc]] LayoutLMv2Model
    - forward

## LayoutLMv2ForSequenceClassification

[[autodoc]] LayoutLMv2ForSequenceClassification

## LayoutLMv2ForTokenClassification

[[autodoc]] LayoutLMv2ForTokenClassification

## LayoutLMv2ForQuestionAnswering

[[autodoc]] LayoutLMv2ForQuestionAnswering


key: _node_content
string_value:
{"id_": "69bb0690-7085-47a1-9154-1113abeea30e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/retribert", "title": "RetriBERT", "tokens": 462, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "79060bf7-6d83-45d0-90b4-035b22c8d9f1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/retribert", "title": "RetriBERT", "tokens": 462, "retrieve_doc": true, "source": "transformers"}, "hash": "e22becdb82c278b17d0311781b3b1504689ce983bc1016c950ea6ac26e3db3e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1127, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RetriBERT

<Tip warning={true}>

This model is in maintenance mode only, so we won't accept any new PRs changing its code.

If you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.
You can do so by running the following command: `pip install -U transformers==4.30.0`.

</Tip>

## Overview

The RetriBERT model was proposed in the blog post [Explain Anything Like I'm Five: A Model for Open Domain Long Form
Question Answering](https://yjernite.github.io/lfqa.html). RetriBERT is a small model that uses either a single or
pair of BERT encoders with lower-dimension projection for dense semantic indexing of text.

This model was contributed by [yjernite](https://huggingface.co/yjernite). Code to train and use the model can be
found [here](https://github.com/huggingface/transformers/tree/main/examples/research-projects/distillation).


## RetriBertConfig

[[autodoc]] RetriBertConfig

## RetriBertTokenizer

[[autodoc]] RetriBertTokenizer

## RetriBertTokenizerFast

[[autodoc]] RetriBertTokenizerFast

## RetriBertModel

[[autodoc]] RetriBertModel
    - forward


key: _node_content
string_value:
{"id_": "4edc5c68-1c06-4b72-8b15-abf5f3717006", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f87aa01-19f5-4dfd-aabd-79170a2855a4", "node_type": "1", "metadata": {}, "hash": "fb46d642bf545c8182efb842fe820fc6c157f1937f480a046cbd2d6774292712", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3230, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Bark

## Overview

Bark is a transformer-based text-to-speech model proposed by Suno AI in [suno-ai/bark](https://github.com/suno-ai/bark).

Bark is made of 4 main models:

- [`BarkSemanticModel`] (also referred to as the 'text' model): a causal auto-regressive transformer model that takes as input tokenized text, and predicts semantic text tokens that capture the meaning of the text.
- [`BarkCoarseModel`] (also referred to as the 'coarse acoustics' model): a causal autoregressive transformer, that takes as input the results of the [`BarkSemanticModel`] model. It aims at predicting the first two audio codebooks necessary for EnCodec.
- [`BarkFineModel`] (the 'fine acoustics' model), this time a non-causal autoencoder transformer, which iteratively predicts the last codebooks based on the sum of the previous codebooks embeddings.
- having predicted all the codebook channels from the [`EncodecModel`], Bark uses it to decode the output audio array.

It should be noted that each of the first three modules can support conditional speaker embeddings to condition the output sound according to specific predefined voice.

This model was contributed by [Yoach Lacombe (ylacombe)](https://huggingface.co/ylacombe) and [Sanchit Gandhi (sanchit-gandhi)](https://github.com/sanchit-gandhi).
The original code can be found [here](https://github.com/suno-ai/bark).

### Optimizing Bark

Bark can be optimized with just a few extra lines of code, which **significantly reduces its memory footprint** and **accelerates inference**.

#### Using half-precision

You can speed up inference and reduce memory footprint by 50% simply by loading the model in half-precision.

```python
from transformers import BarkModel
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16).to(device)
```

#### Using CPU offload

As mentioned above, Bark is made up of 4 sub-models, which are called up sequentially during audio generation. In other words, while one sub-model is in use, the other sub-models are idle.

If you're using a CUDA device, a simple solution to benefit from an 80% reduction in memory footprint is to offload the submodels from GPU to CPU when they're idle. This operation is called *CPU offloading*. You can use it with one line of code as follows:

```python
model.enable_cpu_offload()
```

Note that 🤗 Accelerate must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/accelerate/basic_tutorials/install)

#### Using Better Transformer

Better Transformer is an 🤗 Optimum feature that performs kernel fusion under the hood. You can gain 20% to 30% in speed with zero performance degradation. It only requires one line of code to export the model to 🤗 Better Transformer:

```python
model =  model.to_bettertransformer()
```

Note that 🤗 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)

#### Using Flash Attention 2

Flash Attention 2 is an even faster, optimized version of the previous optimization.

##### Installation 

First, check whether your hardware is compatible with Flash Attention 2.


key: _node_content
string_value:
{"id_": "6f87aa01-19f5-4dfd-aabd-79170a2855a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4edc5c68-1c06-4b72-8b15-abf5f3717006", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "5c4e160636eb0fdaa9b1c6aea0b758e554e4672797e511c71a9c485f7759856f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5526658e-858c-4b78-8c51-9cd9f2787469", "node_type": "1", "metadata": {}, "hash": "4a2a8f083b7c71018ca31ca093bf9928cf9f957002a454970c96f09db8c09131", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1535, "end_char_idx": 4457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Using half-precision

You can speed up inference and reduce memory footprint by 50% simply by loading the model in half-precision.

```python
from transformers import BarkModel
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16).to(device)
```

#### Using CPU offload

As mentioned above, Bark is made up of 4 sub-models, which are called up sequentially during audio generation. In other words, while one sub-model is in use, the other sub-models are idle.

If you're using a CUDA device, a simple solution to benefit from an 80% reduction in memory footprint is to offload the submodels from GPU to CPU when they're idle. This operation is called *CPU offloading*. You can use it with one line of code as follows:

```python
model.enable_cpu_offload()
```

Note that 🤗 Accelerate must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/accelerate/basic_tutorials/install)

#### Using Better Transformer

Better Transformer is an 🤗 Optimum feature that performs kernel fusion under the hood. You can gain 20% to 30% in speed with zero performance degradation. It only requires one line of code to export the model to 🤗 Better Transformer:

```python
model =  model.to_bettertransformer()
```

Note that 🤗 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)

#### Using Flash Attention 2

Flash Attention 2 is an even faster, optimized version of the previous optimization.

##### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```


##### Usage

To load a model using Flash Attention 2, we can pass the `attn_implementation="flash_attention_2"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
```

##### Performance comparison


key: _node_content
string_value:
{"id_": "5526658e-858c-4b78-8c51-9cd9f2787469", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f87aa01-19f5-4dfd-aabd-79170a2855a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "9496910fd08e3947854fa3dd7c18cc72887c0a9ab34beada93f1307fd34f5f21", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05404693-f199-4e83-a337-c0092096dbe5", "node_type": "1", "metadata": {}, "hash": "e47fc7a09883719e5c019374ab407b628b1bcd859cddda56c88f20f673fccbd5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2744, "end_char_idx": 5495, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It only requires one line of code to export the model to 🤗 Better Transformer:

```python
model =  model.to_bettertransformer()
```

Note that 🤗 Optimum must be installed before using this feature. [Here's how to install it.](https://huggingface.co/docs/optimum/installation)

#### Using Flash Attention 2

Flash Attention 2 is an even faster, optimized version of the previous optimization.

##### Installation 

First, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).

Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```


##### Usage

To load a model using Flash Attention 2, we can pass the `attn_implementation="flash_attention_2"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
```

##### Performance comparison


The following diagram shows the latency for the native attention implementation (no optimisation) against Better Transformer and Flash Attention 2. In all cases, we generate 400 semantic tokens on a 40GB A100 GPU with PyTorch 2.1. Flash Attention 2 is also consistently faster than Better Transformer, and its performance improves even more as batch sizes increase:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ylacombe/benchmark-comparison/resolve/main/Bark%20Optimization%20Benchmark.png">
</div>

To put this into perspective, on an NVIDIA A100 and when generating 400 semantic tokens with a batch size of 16, you can get 17 times the [throughput](https://huggingface.co/blog/optimizing-bark#throughput) and still be 2 seconds faster than generating sentences one by one with the native model implementation. In other words, all the samples will be generated 17 times faster.

At batch size 8, on an NVIDIA A100, Flash Attention 2 is also 10% faster than Better Transformer, and at batch size 16, 25%.


key: _node_content
string_value:
{"id_": "05404693-f199-4e83-a337-c0092096dbe5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5526658e-858c-4b78-8c51-9cd9f2787469", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "2cb92c43baff8cdd69ec61a16e32c25a6d1b7cbb4227a60e4f8e814b1895688d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "062af563-5753-4101-a5a0-ef087b71a5ef", "node_type": "1", "metadata": {}, "hash": "3ec107b5b570974fe4a806e8de33ce2b70d0cea59aa5d31c6e08ca3cfcba2ef4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3839, "end_char_idx": 7658, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
##### Usage

To load a model using Flash Attention 2, we can pass the `attn_implementation="flash_attention_2"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:

```python
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)
```

##### Performance comparison


The following diagram shows the latency for the native attention implementation (no optimisation) against Better Transformer and Flash Attention 2. In all cases, we generate 400 semantic tokens on a 40GB A100 GPU with PyTorch 2.1. Flash Attention 2 is also consistently faster than Better Transformer, and its performance improves even more as batch sizes increase:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ylacombe/benchmark-comparison/resolve/main/Bark%20Optimization%20Benchmark.png">
</div>

To put this into perspective, on an NVIDIA A100 and when generating 400 semantic tokens with a batch size of 16, you can get 17 times the [throughput](https://huggingface.co/blog/optimizing-bark#throughput) and still be 2 seconds faster than generating sentences one by one with the native model implementation. In other words, all the samples will be generated 17 times faster.

At batch size 8, on an NVIDIA A100, Flash Attention 2 is also 10% faster than Better Transformer, and at batch size 16, 25%.


#### Combining optimization techniques

You can combine optimization techniques, and use CPU offload, half-precision and Flash Attention 2 (or 🤗 Better Transformer) all at once.

```python
from transformers import BarkModel
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

# load in fp16 and use Flash Attention 2
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16, attn_implementation="flash_attention_2").to(device)

# enable CPU offload
model.enable_cpu_offload()
```

Find out more on inference optimization techniques [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one).

### Usage tips

Suno offers a library of voice presets in a number of languages [here](https://suno-ai.notion.site/8b8e8749ed514b0cbf3f699013548683?v=bc67cff786b04b50b3ceb756fd05f68c).
These presets are also uploaded in the hub [here](https://huggingface.co/suno/bark-small/tree/main/speaker_embeddings) or [here](https://huggingface.co/suno/bark/tree/main/speaker_embeddings).

```python
>>> from transformers import AutoProcessor, BarkModel

>>> processor = AutoProcessor.from_pretrained("suno/bark")
>>> model = BarkModel.from_pretrained("suno/bark")

>>> voice_preset = "v2/en_speaker_6"

>>> inputs = processor("Hello, my dog is cute", voice_preset=voice_preset)

>>> audio_array = model.generate(**inputs)
>>> audio_array = audio_array.cpu().numpy().squeeze()
```

Bark can generate highly realistic, **multilingual** speech as well as other audio - including music, background noise and simple sound effects. 

```python
>>> # Multilingual speech - simplified Chinese
>>> inputs = processor("惊人的！我会说中文")

>>> # Multilingual speech - French - let's use a voice_preset as well
>>> inputs = processor("Incroyable! Je peux générer du son.", voice_preset="fr_speaker_5")

>>> # Bark can also generate music. You can help it out by adding music notes around your lyrics.
>>> inputs = processor("♪ Hello, my dog is cute ♪")

>>> audio_array = model.generate(**inputs)
>>> audio_array = audio_array.cpu().numpy().squeeze()
```

The model can also produce **nonverbal communications** like laughing, sighing and crying.


key: _node_content
string_value:
{"id_": "062af563-5753-4101-a5a0-ef087b71a5ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05404693-f199-4e83-a337-c0092096dbe5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers"}, "hash": "817762c7b856ebfc08f2649840eedecbe664779483f7169b62e348e54dd74a3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7661, "end_char_idx": 8890, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> # Adding non-speech cues to the input text
>>> inputs = processor("Hello uh ... [clears throat], my dog is cute [laughter]")

>>> audio_array = model.generate(**inputs)
>>> audio_array = audio_array.cpu().numpy().squeeze()
```

To save the audio, simply take the sample rate from the model config and some scipy utility:

```python
>>> from scipy.io.wavfile import write as write_wav

>>> # save audio to disk, but first take the sample rate from the model config
>>> sample_rate = model.generation_config.sample_rate
>>> write_wav("bark_generation.wav", sample_rate, audio_array)
```

## BarkConfig

[[autodoc]] BarkConfig
    - all

## BarkProcessor

[[autodoc]] BarkProcessor
    - all
    - __call__

## BarkModel

[[autodoc]] BarkModel
    - generate
    - enable_cpu_offload

## BarkSemanticModel

[[autodoc]] BarkSemanticModel
    - forward

## BarkCoarseModel

[[autodoc]] BarkCoarseModel
    - forward

## BarkFineModel

[[autodoc]] BarkFineModel
    - forward

## BarkCausalModel

[[autodoc]] BarkCausalModel
    - forward

## BarkCoarseConfig

[[autodoc]] BarkCoarseConfig
    - all

## BarkFineConfig

[[autodoc]] BarkFineConfig
    - all

## BarkSemanticConfig

[[autodoc]] BarkSemanticConfig
    - all


key: _node_content
string_value:
{"id_": "a601726e-3fb7-437a-bc5a-f344f18e42c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers"}, "hash": "c132dc5980022b1c3c3eb0cccd19de9d7dd59841cb2e25ce19d9b0a3e0bce6e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "32eade85-ccda-4557-aaf1-aa68c15b6550", "node_type": "1", "metadata": {}, "hash": "623b37112bb4f8c2164042414adf8ad56ea8be22598a1cabb170134a86e6ac2c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3267, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ErnieM

<Tip warning={true}>

This model is in maintenance mode only, we don't accept any new PRs changing its code.
If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.
You can do so by running the following command: `pip install -U transformers==4.40.2`.

</Tip>

## Overview

The ErnieM model was proposed in [ERNIE-M: Enhanced Multilingual Representation by Aligning
Cross-lingual Semantics with Monolingual Corpora](https://arxiv.org/abs/2012.15674)  by Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun,
Hao Tian, Hua Wu, Haifeng Wang.

The abstract from the paper is the following:

*Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for lowresource languages. In this paper, we propose ERNIE-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks.*
This model was contributed by [Susnato Dhar](https://huggingface.co/susnato). The original code can be found [here](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers/ernie_m).


## Usage tips

- Ernie-M is a BERT-like model so it is a stacked Transformer Encoder.
- Instead of using MaskedLM for pretraining (like BERT) the authors used two novel techniques: `Cross-attention Masked Language Modeling` and `Back-translation Masked Language Modeling`. For now these two LMHead objectives are not implemented here.
- It is a multilingual language model.
- Next Sentence Prediction was not used in pretraining process.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Multiple choice task guide](../tasks/multiple_choice)

## ErnieMConfig

[[autodoc]] ErnieMConfig


## ErnieMTokenizer

[[autodoc]] ErnieMTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary


## ErnieMModel

[[autodoc]] ErnieMModel
    - forward

## ErnieMForSequenceClassification

[[autodoc]] ErnieMForSequenceClassification
    - forward


## ErnieMForMultipleChoice

[[autodoc]] ErnieMForMultipleChoice
    - forward


## ErnieMForTokenClassification

[[autodoc]] ErnieMForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "32eade85-ccda-4557-aaf1-aa68c15b6550", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers"}, "hash": "c132dc5980022b1c3c3eb0cccd19de9d7dd59841cb2e25ce19d9b0a3e0bce6e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a601726e-3fb7-437a-bc5a-f344f18e42c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers"}, "hash": "4993d7f8dc72344bbba71ed3a904493d30cd2e7c1af5d98bd8997e45f73c3814", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2011, "end_char_idx": 3446, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips

- Ernie-M is a BERT-like model so it is a stacked Transformer Encoder.
- Instead of using MaskedLM for pretraining (like BERT) the authors used two novel techniques: `Cross-attention Masked Language Modeling` and `Back-translation Masked Language Modeling`. For now these two LMHead objectives are not implemented here.
- It is a multilingual language model.
- Next Sentence Prediction was not used in pretraining process.

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Multiple choice task guide](../tasks/multiple_choice)

## ErnieMConfig

[[autodoc]] ErnieMConfig


## ErnieMTokenizer

[[autodoc]] ErnieMTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary


## ErnieMModel

[[autodoc]] ErnieMModel
    - forward

## ErnieMForSequenceClassification

[[autodoc]] ErnieMForSequenceClassification
    - forward


## ErnieMForMultipleChoice

[[autodoc]] ErnieMForMultipleChoice
    - forward


## ErnieMForTokenClassification

[[autodoc]] ErnieMForTokenClassification
    - forward


## ErnieMForQuestionAnswering

[[autodoc]] ErnieMForQuestionAnswering
    - forward

## ErnieMForInformationExtraction

[[autodoc]] ErnieMForInformationExtraction
    - forward


key: _node_content
string_value:
{"id_": "18d78e40-87b9-402d-a50b-755fc8163010", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers"}, "hash": "9710c0628b0c252fe9fb1c339e373efe622fa35a4dc4f0ef0decb79e1c84ea4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "84b5362e-1d63-4908-b892-9c14bfd0bc29", "node_type": "1", "metadata": {}, "hash": "050bdc5091aae5f3ec9e79e94af8e9a1e3d00366402b636df996b49b515483db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3280, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SegGPT

## Overview

The SegGPT model was proposed in [SegGPT: Segmenting Everything In Context](https://arxiv.org/abs/2304.03284) by Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, Tiejun Huang. SegGPT employs a decoder-only Transformer that can generate a segmentation mask given an input image, a prompt image and its corresponding prompt mask. The model achieves remarkable one-shot results with 56.1 mIoU on COCO-20 and 85.6 mIoU on FSS-1000.

The abstract from the paper is the following:

*We present SegGPT, a generalist model for segmenting everything in context. We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images. The training of SegGPT is formulated as an in-context coloring problem with random color mapping for each data sample. The objective is to accomplish diverse tasks according to the context, rather than relying on specific colors. After training, SegGPT can perform arbitrary segmentation tasks in images or videos via in-context inference, such as object instance, stuff, part, contour, and text. SegGPT is evaluated on a broad range of tasks, including few-shot semantic segmentation, video object segmentation, semantic segmentation, and panoptic segmentation. Our results show strong capabilities in segmenting in-domain and out-of*

Tips:
- One can use [`SegGptImageProcessor`] to prepare image input, prompt and mask to the model.
- One can either use segmentation maps or RGB images as prompt masks. If using the latter make sure to set `do_convert_rgb=False` in the `preprocess` method.
- It's highly advisable to pass `num_labels` when using `segmetantion_maps` (not considering background) during preprocessing and postprocessing with [`SegGptImageProcessor`] for your use case.
- When doing inference with [`SegGptForImageSegmentation`] if your `batch_size` is greater than 1 you can use feature ensemble across your images by passing `feature_ensemble=True` in the forward method.

Here's how to use the model for one-shot semantic segmentation:

```python
import torch
from datasets import load_dataset
from transformers import SegGptImageProcessor, SegGptForImageSegmentation

checkpoint = "BAAI/seggpt-vit-large"
image_processor = SegGptImageProcessor.from_pretrained(checkpoint)
model = SegGptForImageSegmentation.from_pretrained(checkpoint)

dataset_id = "EduardoPacheco/FoodSeg103"
ds = load_dataset(dataset_id, split="train")
# Number of labels in FoodSeg103 (not including background)
num_labels = 103

image_input = ds[4]["image"]
ground_truth = ds[4]["label"]
image_prompt = ds[29]["image"]
mask_prompt = ds[29]["label"]

inputs = image_processor(
    images=image_input, 
    prompt_images=image_prompt,
    segmentation_maps=mask_prompt, 
    num_labels=num_labels,
    return_tensors="pt"
)

with torch.no_grad():
    outputs = model(**inputs)

target_sizes = [image_input.size[::-1]]
mask = image_processor.post_process_semantic_segmentation(outputs, target_sizes, num_labels=num_labels)[0]
```

This model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco).
The original code can be found [here]([(https://github.com/baaivision/Painter/tree/main)).


key: _node_content
string_value:
{"id_": "84b5362e-1d63-4908-b892-9c14bfd0bc29", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers"}, "hash": "9710c0628b0c252fe9fb1c339e373efe622fa35a4dc4f0ef0decb79e1c84ea4a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "18d78e40-87b9-402d-a50b-755fc8163010", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers"}, "hash": "88329ac691b57817d5a4fe064260a73ce0f3e8b487c5e6907c4f2f2db099b812", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3283, "end_char_idx": 3581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## SegGptConfig

[[autodoc]] SegGptConfig

## SegGptImageProcessor

[[autodoc]] SegGptImageProcessor
    - preprocess
    - post_process_semantic_segmentation

## SegGptModel

[[autodoc]] SegGptModel
    - forward

## SegGptForImageSegmentation

[[autodoc]] SegGptForImageSegmentation
    - forward


key: _node_content
string_value:
{"id_": "a2174e29-3665-499a-aaef-bcb1632f29d8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "hash": "def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "412a0a25-9857-4292-a4ff-e15cbb8750c1", "node_type": "1", "metadata": {}, "hash": "1102cfabdfdb4a0bb54c0fe097f018c19a4477b104572e2c3f9197f8e9765d7e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2641, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# FastSpeech2Conformer

## Overview

The FastSpeech2Conformer model was proposed with the paper [Recent Developments On Espnet Toolkit Boosted By Conformer](https://arxiv.org/abs/2010.13956) by Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang.

The abstract from the original FastSpeech2 paper is the following:

*Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.*

This model was contributed by [Connor Henderson](https://huggingface.co/connor-henderson). The original code can be found [here](https://github.com/espnet/espnet/blob/master/espnet2/tts/fastspeech2/fastspeech2.py).


key: _node_content
string_value:
{"id_": "412a0a25-9857-4292-a4ff-e15cbb8750c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "hash": "def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2174e29-3665-499a-aaef-bcb1632f29d8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "hash": "e13746a38df065a33fb502fdbe82b50b75495a593e1121047d344275b85e5680", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "13469cb5-9b57-4354-8f9e-1f0e5b818d04", "node_type": "1", "metadata": {}, "hash": "b54c25d46b697cf2fef25d9b174c94ae3029eff0ee79f865f65a64ece432c581", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2644, "end_char_idx": 5598, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## 🤗 Model Architecture
FastSpeech2's general structure with a Mel-spectrogram decoder was implemented, and the traditional transformer blocks were replaced with conformer blocks as done in the ESPnet library.

#### FastSpeech2 Model Architecture
![FastSpeech2 Model Architecture](https://www.microsoft.com/en-us/research/uploads/prod/2021/04/fastspeech2-1.png)

#### Conformer Blocks
![Conformer Blocks](https://www.researchgate.net/profile/Hirofumi-Inaguma-2/publication/344911155/figure/fig2/AS:951455406108673@1603856054097/An-overview-of-Conformer-block.png)

#### Convolution Module
![Convolution Module](https://d3i71xaburhd42.cloudfront.net/8809d0732f6147d4ad9218c8f9b20227c837a746/2-Figure1-1.png)

## 🤗 Transformers Usage

You can run FastSpeech2Conformer locally with the 🤗 Transformers library.

1. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers), g2p-en:

```bash
pip install --upgrade pip
pip install --upgrade transformers g2p-en
```

2. Run inference via the Transformers modelling code with the model and hifigan separately

```python

from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerModel, FastSpeech2ConformerHifiGan
import soundfile as sf

tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer")
inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt")
input_ids = inputs["input_ids"]

model = FastSpeech2ConformerModel.from_pretrained("espnet/fastspeech2_conformer")
output_dict = model(input_ids, return_dict=True)
spectrogram = output_dict["spectrogram"]

hifigan = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan")
waveform = hifigan(spectrogram)

sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)
```

3. Run inference via the Transformers modelling code with the model and hifigan combined

```python
from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan
import soundfile as sf

tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer")
inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt")
input_ids = inputs["input_ids"]

model = FastSpeech2ConformerWithHifiGan.from_pretrained("espnet/fastspeech2_conformer_with_hifigan")
output_dict = model(input_ids, return_dict=True)
waveform = output_dict["waveform"]

sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)
```

4. Run inference with a pipeline and specify which vocoder to use
```python
from transformers import pipeline, FastSpeech2ConformerHifiGan
import soundfile as sf

vocoder = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan")
synthesiser = pipeline(model="espnet/fastspeech2_conformer", vocoder=vocoder)

speech = synthesiser("Hello, my dog is cooler than you!")

sf.write("speech.wav", speech["audio"].squeeze(), samplerate=speech["sampling_rate"])
```


key: _node_content
string_value:
{"id_": "13469cb5-9b57-4354-8f9e-1f0e5b818d04", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "hash": "def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "412a0a25-9857-4292-a4ff-e15cbb8750c1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers"}, "hash": "0cac8dbac6f52a98e686d55399f44368669e071cba661a3dd9c10cbe5e66f344", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5601, "end_char_idx": 6259, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## FastSpeech2ConformerConfig

[[autodoc]] FastSpeech2ConformerConfig

## FastSpeech2ConformerHifiGanConfig

[[autodoc]] FastSpeech2ConformerHifiGanConfig

## FastSpeech2ConformerWithHifiGanConfig

[[autodoc]] FastSpeech2ConformerWithHifiGanConfig

## FastSpeech2ConformerTokenizer

[[autodoc]] FastSpeech2ConformerTokenizer
    - __call__
    - save_vocabulary
    - decode
    - batch_decode

## FastSpeech2ConformerModel

[[autodoc]] FastSpeech2ConformerModel
    - forward

## FastSpeech2ConformerHifiGan

[[autodoc]] FastSpeech2ConformerHifiGan
    - forward

## FastSpeech2ConformerWithHifiGan

[[autodoc]] FastSpeech2ConformerWithHifiGan
    - forward


key: _node_content
string_value:
{"id_": "a4e64056-3c94-466a-bdde-cd36297fbd46", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers"}, "hash": "40a004e7c40d19e666beb2724c740fe98f61a3bdcd676171eff04e4a0d65d538", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "46f06ca8-18fe-41dd-a5a3-b0b4ce484ba9", "node_type": "1", "metadata": {}, "hash": "a9ca49fb8a63ba71ce65d371bfd3ec931e89e75c573411baa4ba7bc3d15088ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3076, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# X-CLIP

## Overview

The X-CLIP model was proposed in [Expanding Language-Image Pretrained Models for General Video Recognition](https://arxiv.org/abs/2208.02816) by Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling.
X-CLIP is a minimal extension of [CLIP](clip) for video. The model consists of a text encoder, a cross-frame vision encoder, a multi-frame integration Transformer, and a video-specific prompt generator.

The abstract from the paper is the following:

*Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable "zero-shot" generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.*

Tips:

- Usage of X-CLIP is identical to [CLIP](clip).

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png"
alt="drawing" width="600"/>

<small> X-CLIP architecture. Taken from the <a href="https://arxiv.org/abs/2208.02816">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/VideoX/tree/master/X-CLIP).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with X-CLIP.

- Demo notebooks for X-CLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "46f06ca8-18fe-41dd-a5a3-b0b4ce484ba9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers"}, "hash": "40a004e7c40d19e666beb2724c740fe98f61a3bdcd676171eff04e4a0d65d538", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4e64056-3c94-466a-bdde-cd36297fbd46", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers"}, "hash": "fc17b184a9f7f5dccb3f49c342e4d3265c05df0faa5a5bad7142aa4822097b80", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1488, "end_char_idx": 3525, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.*

Tips:

- Usage of X-CLIP is identical to [CLIP](clip).

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png"
alt="drawing" width="600"/>

<small> X-CLIP architecture. Taken from the <a href="https://arxiv.org/abs/2208.02816">original paper.</a> </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/microsoft/VideoX/tree/master/X-CLIP).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with X-CLIP.

- Demo notebooks for X-CLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## XCLIPProcessor

[[autodoc]] XCLIPProcessor

## XCLIPConfig

[[autodoc]] XCLIPConfig
    - from_text_vision_configs

## XCLIPTextConfig

[[autodoc]] XCLIPTextConfig

## XCLIPVisionConfig

[[autodoc]] XCLIPVisionConfig

## XCLIPModel

[[autodoc]] XCLIPModel
    - forward
    - get_text_features
    - get_video_features

## XCLIPTextModel

[[autodoc]] XCLIPTextModel
    - forward

## XCLIPVisionModel

[[autodoc]] XCLIPVisionModel
    - forward


key: _node_content
string_value:
{"id_": "1a08bd05-7ae3-461c-b26d-0d2e7eb0f9d3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "hash": "79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c5c17d0-dbe2-4aae-849c-3a761304a4e9", "node_type": "1", "metadata": {}, "hash": "74dbb02ca2c248bbe5608882fd2c92217fb1857e165ac137047494e4e94ade01", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3001, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VideoMAE

## Overview

The VideoMAE model was proposed in [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://arxiv.org/abs/2203.12602) by Zhan Tong, Yibing Song, Jue Wang, Limin Wang.
VideoMAE extends masked auto encoders ([MAE](vit_mae)) to video, claiming state-of-the-art performance on several video classification benchmarks.

The abstract from the paper is the following:

*Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking and reconstruction. These simple designs turn out to be effective for overcoming information leakage caused by the temporal correlation during video reconstruction. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. This is partially ascribed to the challenging task of video reconstruction to enforce high-level structure learning. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videomae_architecture.jpeg"
alt="drawing" width="600"/>

<small> VideoMAE pre-training. Taken from the <a href="https://arxiv.org/abs/2203.12602">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/MCG-NJU/VideoMAE).

## Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.


key: _node_content
string_value:
{"id_": "3c5c17d0-dbe2-4aae-849c-3a761304a4e9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "hash": "79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1a08bd05-7ae3-461c-b26d-0d2e7eb0f9d3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "hash": "c8c73508872c436c77f109629f644f2dee085f32fe4b5c9769548c78c65b936b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c619bd65-ef11-42a5-b8e9-68e3c7f75c65", "node_type": "1", "metadata": {}, "hash": "26daa989fc42d04257acb5afb63fce97a5306762c8d57c0b8926d6290f19a639", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1542, "end_char_idx": 4683, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videomae_architecture.jpeg"
alt="drawing" width="600"/>

<small> VideoMAE pre-training. Taken from the <a href="https://arxiv.org/abs/2203.12602">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr).
The original code can be found [here](https://github.com/MCG-NJU/VideoMAE).

## Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import VideoMAEForVideoClassification
model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MCG-NJU/videomae-base-finetuned-kinetics` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        37 |                                        10 |                      3.7  |
|            2 |                                        24 |                                        18 |                      1.33 |
|            4 |                                        43 |                                        32 |                      1.34 |
|            8 |                                        84 |                                        60 |                      1.4  |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with VideoMAE. If
you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll
review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "c619bd65-ef11-42a5-b8e9-68e3c7f75c65", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "hash": "79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c5c17d0-dbe2-4aae-849c-3a761304a4e9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers"}, "hash": "0197d4fba3265d40096d73ab77e5df56b831a3af7a01451f8bf95be4761dfbac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3003, "end_char_idx": 5655, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```
from transformers import VideoMAEForVideoClassification
model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MCG-NJU/videomae-base-finetuned-kinetics` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                        37 |                                        10 |                      3.7  |
|            2 |                                        24 |                                        18 |                      1.33 |
|            4 |                                        43 |                                        32 |                      1.34 |
|            8 |                                        84 |                                        60 |                      1.4  |

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with VideoMAE. If
you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll
review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

**Video classification**
- [A notebook](https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb) that shows how
to fine-tune a VideoMAE model on a custom dataset.
- [Video classification task guide](../tasks/video_classification)
- [A 🤗 Space](https://huggingface.co/spaces/sayakpaul/video-classification-ucf101-subset) showing how to perform inference with a video classification model.

## VideoMAEConfig

[[autodoc]] VideoMAEConfig

## VideoMAEFeatureExtractor

[[autodoc]] VideoMAEFeatureExtractor
    - __call__

## VideoMAEImageProcessor

[[autodoc]] VideoMAEImageProcessor
    - preprocess

## VideoMAEModel

[[autodoc]] VideoMAEModel
    - forward

## VideoMAEForPreTraining

`VideoMAEForPreTraining` includes the decoder on top for self-supervised pre-training.

[[autodoc]] transformers.VideoMAEForPreTraining
    - forward

## VideoMAEForVideoClassification

[[autodoc]] transformers.VideoMAEForVideoClassification
    - forward


key: _node_content
string_value:
{"id_": "e2ec2ff4-a3f5-419e-817a-5b3626fc78a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b5eb4a2d-b672-4143-8640-3af2462eec36", "node_type": "1", "metadata": {}, "hash": "f208690bb0a8145af4cd177ceefb3138f224890000facd968d3c493ff40d5ea9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3328, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Vision Transformer (ViT)

## Overview

The Vision Transformer (ViT) model was proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition
at Scale](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk
Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob
Uszkoreit, Neil Houlsby. It's the first paper that successfully trains a Transformer encoder on ImageNet, attaining
very good results compared to familiar convolutional architectures.

The abstract from the paper is the following:

*While the Transformer architecture has become the de-facto standard for natural language processing tasks, its
applications to computer vision remain limited. In vision, attention is either applied in conjunction with
convolutional networks, or used to replace certain components of convolutional networks while keeping their overall
structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to
sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of
data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),
Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring
substantially fewer computational resources to train.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg"
alt="drawing" width="600"/>

<small> ViT architecture. Taken from the <a href="https://arxiv.org/abs/2010.11929">original paper.</a> </small>

Following the original Vision Transformer, some follow-up works have been made:

- [DeiT](deit) (Data-efficient Image Transformers) by Facebook AI. DeiT models are distilled vision transformers.
  The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [`ViTModel`] or
  [`ViTForImageClassification`]. There are 4 variants available (in 3 different sizes): *facebook/deit-tiny-patch16-224*,
  *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and *facebook/deit-base-patch16-384*. Note that one should
  use [`DeiTImageProcessor`] in order to prepare images for the model.

- [BEiT](beit) (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained
  vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a VQ-VAE.

- DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using
  the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting
  objects, without having ever been trained to do so. DINO checkpoints can be found on the [hub](https://huggingface.co/models?other=dino).

- [MAE](vit_mae) (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion
  (75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms
  supervised pre-training after fine-tuning.


key: _node_content
string_value:
{"id_": "b5eb4a2d-b672-4143-8640-3af2462eec36", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2ec2ff4-a3f5-419e-817a-5b3626fc78a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "23a8450c56f3d3c4fbaa0dca78fccb707024934bb661b690692bc71780c076fc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dcc57756-49e6-482a-be7d-7d3406ec0728", "node_type": "1", "metadata": {}, "hash": "0f3c80007d7fa609bd8b70a9b5d01f567f2249d9ee639c32bfa5762877ae9c14", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1900, "end_char_idx": 5083, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
DeiT models are distilled vision transformers.
  The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [`ViTModel`] or
  [`ViTForImageClassification`]. There are 4 variants available (in 3 different sizes): *facebook/deit-tiny-patch16-224*,
  *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and *facebook/deit-base-patch16-384*. Note that one should
  use [`DeiTImageProcessor`] in order to prepare images for the model.

- [BEiT](beit) (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained
  vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a VQ-VAE.

- DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using
  the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting
  objects, without having ever been trained to do so. DINO checkpoints can be found on the [hub](https://huggingface.co/models?other=dino).

- [MAE](vit_mae) (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion
  (75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms
  supervised pre-training after fine-tuning.

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be
found [here](https://github.com/google-research/vision_transformer).

Note that we converted the weights from Ross Wightman's [timm library](https://github.com/rwightman/pytorch-image-models),
who already converted the weights from JAX to PyTorch. Credits go to him!

## Usage tips

- To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,
  which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can be
  used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of
  vectors to a standard Transformer encoder.
- As the Vision Transformer expects each image to be of the same size (resolution), one can use
  [`ViTImageProcessor`] to resize (or rescale) and normalize images for the model.
- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of
  each checkpoint. For example, `google/vit-base-patch16-224` refers to a base-sized architecture with patch
  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=vit).
- The available checkpoints are either (1) pre-trained on [ImageNet-21k](http://www.image-net.org/) (a collection of
  14 million images and 21k classes) only, or (2) also fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million
  images and 1,000 classes).
- The Vision Transformer was pre-trained using a resolution of 224x224.


key: _node_content
string_value:
{"id_": "dcc57756-49e6-482a-be7d-7d3406ec0728", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b5eb4a2d-b672-4143-8640-3af2462eec36", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "b7965715fc65dfd16e15c002fd79ecf5931458c52426f6aeb6734d5ac832f20b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7e45117-f6dd-4a51-8cfb-622ea7f19658", "node_type": "1", "metadata": {}, "hash": "f000ddf0efdd7ec24f787f5f3e31fecfb6fd76251f7e04e948cc7652721115a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3513, "end_char_idx": 6673, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that we converted the weights from Ross Wightman's [timm library](https://github.com/rwightman/pytorch-image-models),
who already converted the weights from JAX to PyTorch. Credits go to him!

## Usage tips

- To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,
  which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can be
  used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of
  vectors to a standard Transformer encoder.
- As the Vision Transformer expects each image to be of the same size (resolution), one can use
  [`ViTImageProcessor`] to resize (or rescale) and normalize images for the model.
- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of
  each checkpoint. For example, `google/vit-base-patch16-224` refers to a base-sized architecture with patch
  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=vit).
- The available checkpoints are either (1) pre-trained on [ImageNet-21k](http://www.image-net.org/) (a collection of
  14 million images and 21k classes) only, or (2) also fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million
  images and 1,000 classes).
- The Vision Transformer was pre-trained using a resolution of 224x224. During fine-tuning, it is often beneficial to
  use a higher resolution than pre-training [(Touvron et al., 2019)](https://arxiv.org/abs/1906.06423), [(Kolesnikov
  et al., 2020)](https://arxiv.org/abs/1912.11370). In order to fine-tune at higher resolution, the authors perform
  2D interpolation of the pre-trained position embeddings, according to their location in the original image.
- The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed
  an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked
  language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant
  improvement of 2% to training from scratch, but still 4% behind supervised pre-training.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.


key: _node_content
string_value:
{"id_": "d7e45117-f6dd-4a51-8cfb-622ea7f19658", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dcc57756-49e6-482a-be7d-7d3406ec0728", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "e20ca46aa6aa681bf39b5c915a2a9069f90ab05bb8c01b964cdf9b40826f3987", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ebf77dc0-e7e1-4f52-b8fd-de830060862d", "node_type": "1", "metadata": {}, "hash": "c7d150dd84c9eb6b0fa496bbcd10457e8a7789d2ee87cbd5fb641d2e832843eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5084, "end_char_idx": 8496, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
During fine-tuning, it is often beneficial to
  use a higher resolution than pre-training [(Touvron et al., 2019)](https://arxiv.org/abs/1906.06423), [(Kolesnikov
  et al., 2020)](https://arxiv.org/abs/1912.11370). In order to fine-tune at higher resolution, the authors perform
  2D interpolation of the pre-trained position embeddings, according to their location in the original image.
- The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed
  an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked
  language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant
  improvement of 2% to training from scratch, but still 4% behind supervised pre-training.

### Using Scaled Dot Product Attention (SDPA)

PyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function 
encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the 
[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) 
or the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)
page for more information.

SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set 
`attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

```
from transformers import ViTForImageClassification
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-base-patch16-224` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         7 |                                         6 |                      1.17 |
|            2 |                                         8 |                                         6 |                      1.33 |
|            4 |                                         8 |                                         6 |                      1.33 |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

Demo notebooks regarding inference as well as fine-tuning ViT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer).
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.


key: _node_content
string_value:
{"id_": "ebf77dc0-e7e1-4f52-b8fd-de830060862d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7e45117-f6dd-4a51-8cfb-622ea7f19658", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers"}, "hash": "9b091568d32bc12de1944061c5fce4f5410ea18229b90ee83142fad3634591a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6675, "end_char_idx": 11212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```
from transformers import ViTForImageClassification
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224", attn_implementation="sdpa", torch_dtype=torch.float16)
...
```

For the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).

On a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-base-patch16-224` model, we saw the following speedups during inference.

|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |
|--------------|-------------------------------------------|-------------------------------------------|------------------------------|
|            1 |                                         7 |                                         6 |                      1.17 |
|            2 |                                         8 |                                         6 |                      1.33 |
|            4 |                                         8 |                                         6 |                      1.33 |
|            8 |                                         8 |                                         6 |                      1.33 |

## Resources

Demo notebooks regarding inference as well as fine-tuning ViT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer).
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ViT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

`ViTForImageClassification` is supported by:
<PipelineTag pipeline="image-classification"/>

- A blog post on how to [Fine-Tune ViT for Image Classification with Hugging Face Transformers](https://huggingface.co/blog/fine-tune-vit)
- A blog post on [Image Classification with Hugging Face Transformers and `Keras`](https://www.philschmid.de/image-classification-huggingface-transformers-keras)
- A notebook on [Fine-tuning for Image Classification with Hugging Face Transformers](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb)
- A notebook on how to [Fine-tune the Vision Transformer on CIFAR-10 with the Hugging Face Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb)
- A notebook on how to [Fine-tune the Vision Transformer on CIFAR-10 with PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb)

⚗️ Optimization

- A blog post on how to [Accelerate Vision Transformer (ViT) with Quantization using Optimum](https://www.philschmid.de/optimizing-vision-transformer)

⚡️ Inference

- A notebook on [Quick demo: Vision Transformer (ViT) by Google Brain](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Quick_demo_of_HuggingFace_version_of_Vision_Transformer_inference.ipynb)

🚀 Deploy

- A blog post on [Deploying Tensorflow Vision Models in Hugging Face with TF Serving](https://huggingface.co/blog/tf-serving-vision)
- A blog post on [Deploying Hugging Face ViT on Vertex AI](https://huggingface.co/blog/deploy-vertex-ai)
- A blog post on [Deploying Hugging Face ViT on Kubernetes with TF Serving](https://huggingface.co/blog/deploy-tfserving-kubernetes)

## ViTConfig

[[autodoc]] ViTConfig

## ViTFeatureExtractor

[[autodoc]] ViTFeatureExtractor
    - __call__

## ViTImageProcessor

[[autodoc]] ViTImageProcessor
    - preprocess

## ViTImageProcessorFast

[[autodoc]] ViTImageProcessorFast
    - preprocess

<frameworkcontent>
<pt>

## ViTModel

[[autodoc]] ViTModel
    - forward

## ViTForMaskedImageModeling

[[autodoc]] ViTForMaskedImageModeling
    - forward

## ViTForImageClassification

[[autodoc]] ViTForImageClassification
    - forward

</pt>
<tf>

## TFViTModel

[[autodoc]] TFViTModel
    - call

## TFViTForImageClassification

[[autodoc]] TFViTForImageClassification
    - call

</tf>
<jax>

## FlaxVitModel

[[autodoc]] FlaxViTModel
    - __call__

## FlaxViTForImageClassification

[[autodoc]] FlaxViTForImageClassification
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "6d1590db-f71e-4a9d-bc34-2fbe397fc085", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "hash": "e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b25813ee-df36-4019-bb10-6707a178f8cb", "node_type": "1", "metadata": {}, "hash": "5f6d9d44d2a0a2dc8f7dff3d4017a86ae4e29dfd3a5d2a5baba17cb5056c4053", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3224, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Jamba

## Overview

Jamba is a state-of-the-art, hybrid SSM-Transformer LLM. It is the first production-scale Mamba implementation, which opens up interesting research and application opportunities. While this initial experimentation shows encouraging gains, we expect these to be further enhanced with future optimizations and explorations.

For full details of this model please read the [release blog post](https://www.ai21.com/blog/announcing-jamba).

### Model Details

Jamba is a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and an overall of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.

As depicted in the diagram below, Jamba's architecture features a blocks-and-layers approach that allows Jamba to successfully integrate Transformer and Mamba architectures altogether. Each Jamba block contains either an attention or a Mamba layer, followed by a multi-layer perceptron (MLP), producing an overall ratio of one Transformer layer out of every eight total layers.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/jamba_architecture.png"
alt="drawing" width="600"/>

## Usage

### Presequities

Jamba requires you use `transformers` version 4.39.0 or higher:
```bash
pip install transformers>=4.39.0
```

In order to run optimized Mamba implementations, you first need to install `mamba-ssm` and `causal-conv1d`:
```bash
pip install mamba-ssm causal-conv1d>=1.2.0
```
You also have to have the model on a CUDA device.

You can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.

### Run the model
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1")
tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1")

input_ids = tokenizer("In the recent Super Bowl LVIII,", return_tensors='pt').to(model.device)["input_ids"]

outputs = model.generate(input_ids, max_new_tokens=216)

print(tokenizer.batch_decode(outputs))
# ["<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\n\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\n\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\n\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon.


key: _node_content
string_value:
{"id_": "b25813ee-df36-4019-bb10-6707a178f8cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "hash": "e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d1590db-f71e-4a9d-bc34-2fbe397fc085", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "hash": "33156d6de15c70296dce5b178196027bfee83783965ab70228b683788d48ecde", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae2d17a9-0b31-4f88-a79b-194582372e42", "node_type": "1", "metadata": {}, "hash": "adbba3a1979e614f15bbb5d39c2e92546abab509692eecddb39203c978a41ee5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1611, "end_char_idx": 4816, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.

### Run the model
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1")
tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1")

input_ids = tokenizer("In the recent Super Bowl LVIII,", return_tensors='pt').to(model.device)["input_ids"]

outputs = model.generate(input_ids, max_new_tokens=216)

print(tokenizer.batch_decode(outputs))
# ["<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\n\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\n\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\n\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\n"]
```

<details>
<summary><strong>Loading the model in half precision</strong></summary>

The published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:

```python
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1", torch_dtype=torch.bfloat16)
# you can also use torch_dtype=torch.float16
```

When using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):
```python
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",
                                             torch_dtype=torch.bfloat16,
                                             attn_implementation="flash_attention_2",
                                             device_map="auto")
```

</details>
<details><summary><strong>Load the model in 8-bit</strong></summary>

**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index).


key: _node_content
string_value:
{"id_": "ae2d17a9-0b31-4f88-a79b-194582372e42", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "hash": "e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b25813ee-df36-4019-bb10-6707a178f8cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers"}, "hash": "102366a28ad619aa9e26c486b8e83e3d779cdc33a285b9b82b6fe92c618df6e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3124, "end_char_idx": 5556, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\n"]
```

<details>
<summary><strong>Loading the model in half precision</strong></summary>

The published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:

```python
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1", torch_dtype=torch.bfloat16)
# you can also use torch_dtype=torch.float16
```

When using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):
```python
from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",
                                             torch_dtype=torch.bfloat16,
                                             attn_implementation="flash_attention_2",
                                             device_map="auto")
```

</details>
<details><summary><strong>Load the model in 8-bit</strong></summary>

**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index). In order to not degrade model quality, we recommend to exclude the Mamba blocks from the quantization:

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_skip_modules=["mamba"])
model = AutoModelForCausalLM.from_pretrained(
    "ai21labs/Jamba-v0.1", torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", quantization_config=quantization_config
)
```
</details>

## JambaConfig

[[autodoc]] JambaConfig


## JambaModel

[[autodoc]] JambaModel
    - forward


## JambaForCausalLM

[[autodoc]] JambaForCausalLM
    - forward


## JambaForSequenceClassification

[[autodoc]] transformers.JambaForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "e1b773a9-3cd3-4f69-b746-96b06b0ddf3a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be9a8950-892a-4daa-bc83-ae3d9de68e09", "node_type": "1", "metadata": {}, "hash": "2ba5d0c0df263d545be96b6e21a428d756dcc06322811b8f2408ab72258c6b33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLaMA

## Overview

The LLaMA model was proposed in [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) by Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. It is a collection of foundation language models ranging from 7B to 65B parameters.

The abstract from the paper is the following:

*We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community. *

This model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).

## Usage tips

- Weights for the LLaMA models can be obtained from by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform?usp=send_form)
- After downloading the weights, they will need to be converted to the Hugging Face Transformers format using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

- After conversion, the model and tokenizer can be loaded via:

```python
from transformers import LlamaForCausalLM, LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("/output/path")
model = LlamaForCausalLM.from_pretrained("/output/path")
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.

- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string.


key: _node_content
string_value:
{"id_": "be9a8950-892a-4daa-bc83-ae3d9de68e09", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e1b773a9-3cd3-4f69-b746-96b06b0ddf3a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "af06f8fb49fc311bd236a7d6d21388fa38dfe9d6785b5995f626d16c53f68882", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a", "node_type": "1", "metadata": {}, "hash": "ea5b533443d7a5392adef42f8c712c0e6b5e43149fd5ac37142c44f2f4b5d94a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1846, "end_char_idx": 4707, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The script can be called with the following (example) command:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

- After conversion, the model and tokenizer can be loaded via:

```python
from transformers import LlamaForCausalLM, LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("/output/path")
model = LlamaForCausalLM.from_pretrained("/output/path")
```

Note that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.

- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string.

This model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.


Based on the original LLaMA model, Meta AI has released some follow-up works:

- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. 🌎

<PipelineTag pipeline="question-answering"/>

- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF.


key: _node_content
string_value:
{"id_": "94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be9a8950-892a-4daa-bc83-ae3d9de68e09", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "cb0ca60ff9043520b826d8a931f202e4e38b06d4fdfed2431366ce7814160ffd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d702bd04-353c-4afc-bfb8-8a15e052a03e", "node_type": "1", "metadata": {}, "hash": "f5e831bddeb404184352ad69569fcb49951a541f4cc776139a81b7fb9d6feeec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3192, "end_char_idx": 5795, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.


Based on the original LLaMA model, Meta AI has released some follow-up works:

- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. 🌎

<PipelineTag pipeline="question-answering"/>

- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF.

⚗️ Optimization
- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. 🌎 

⚡️ Inference
- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the 🤗 PEFT library. 🌎 
- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. 🌎

🚀 Deploy
- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the 🤗 PEFT library with intuitive UI. 🌎 
- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker.


key: _node_content
string_value:
{"id_": "d702bd04-353c-4afc-bfb8-8a15e052a03e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers"}, "hash": "e8f7de0fc70065103d7fdfac9857a4378e7fc098e65809756fc1752c6399d4a4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4709, "end_char_idx": 6754, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
⚗️ Optimization
- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. 🌎 

⚡️ Inference
- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the 🤗 PEFT library. 🌎 
- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. 🌎

🚀 Deploy
- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the 🤗 PEFT library with intuitive UI. 🌎 
- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker. 🌎 

## LlamaConfig

[[autodoc]] LlamaConfig

## LlamaTokenizer

[[autodoc]] LlamaTokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## LlamaTokenizerFast

[[autodoc]] LlamaTokenizerFast
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - update_post_processor
    - save_vocabulary

## LlamaModel

[[autodoc]] LlamaModel
    - forward

## LlamaForCausalLM

[[autodoc]] LlamaForCausalLM
    - forward

## LlamaForSequenceClassification

[[autodoc]] LlamaForSequenceClassification
    - forward

## LlamaForQuestionAnswering

[[autodoc]] LlamaForQuestionAnswering
    - forward

## LlamaForTokenClassification

[[autodoc]] LlamaForTokenClassification
    - forward

## FlaxLlamaModel

[[autodoc]] FlaxLlamaModel
    - __call__

## FlaxLlamaForCausalLM

[[autodoc]] FlaxLlamaForCausalLM
    - __call__


key: _node_content
string_value:
{"id_": "2e19f003-df73-4c34-924c-dfac3db8116a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26c13974-a5da-4e90-b232-3f28979c827f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "890818b5-2c8f-420b-a1e4-73da3128a75f", "node_type": "1", "metadata": {}, "hash": "5b1d4e05a9a2ea26c818f5437642aa636fdb448653924674a6de5fc6d7deffba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3383, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLaVa-NeXT-Video

## Overview

The LLaVa-NeXT-Video model was proposed in [LLaVA-NeXT: A Strong Zero-shot Video Understanding Model
](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/) by Yuanhan Zhang, Bo Li, Haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, Chunyuan Li. LLaVa-NeXT-Video improves upon [LLaVa-NeXT](llava_next) by fine-tuning on a mix if video and image dataset thus increasing the model's performance on videos.

[LLaVA-NeXT](llava_next) surprisingly has strong performance in understanding video content in zero-shot fashion with the AnyRes technique that it uses. The AnyRes technique naturally represents a high-resolution image into multiple images. This technique is naturally generalizable to represent videos because videos can be considered as a set of frames (similar to a set of images in LLaVa-NeXT). The current version of LLaVA-NeXT makes use of AnyRes and trains with supervised fine-tuning (SFT) on top of LLaVA-Next on video data to achieves better video understanding capabilities.The model is a current SOTA among open-source models on [VideoMME bench](https://arxiv.org/abs/2405.21075).


The introduction from the blog is the following:

On January 30, 2024, we released LLaVA-NeXT, an open-source Large Multimodal Model (LMM) that has been trained exclusively on text-image data. With the proposed AnyRes technique, it boosts capabilities in reasoning, OCR, and world knowledge, demonstrating remarkable performance across a spectrum of image-based multimodal understanding tasks, and even exceeding Gemini-Pro on several image benchmarks, e.g. MMMU and MathVista.

**In today’s exploration, we delve into the performance of LLaVA-NeXT within the realm of video understanding tasks. We reveal that LLaVA-NeXT surprisingly has strong performance in understanding video content. The current version of LLaVA-NeXT for videos has several improvements:

- Zero-shot video representation capabilities with AnyRes: The AnyRes technique naturally represents a high-resolution image into multiple images that a pre-trained VIT is able to digest, and forms them into a concantenated sequence. This technique is naturally generalizable to represent videos (consisting of multiple frames), allowing the image-only-trained LLaVA-Next model to perform surprisingly well on video tasks. Notably, this is the first time that LMMs show strong zero-shot modality transfer ability.
- Inference with length generalization improves on longer videos. The linear scaling technique enables length generalization, allowing LLaVA-NeXT to effectively handle long-video beyond the limitation of the "max_token_length" of the LLM.
- Strong video understanding ability. (1) LLaVA-Next-Image, which combines the above two techniques, yields superior zero-shot performance than open-source LMMs tuned on videos. (2) LLaVA-Next-Video, further supervised fine-tuning (SFT) LLaVA-Next-Image on video data, achieves better video understanding capabilities compared to LLaVA-Next-Image. (3) LLaVA-Next-Video-DPO, which aligns the model response with AI feedback using direct preference optimization (DPO), showing significant performance boost.
- Efficient deployment and inference with SGLang. It allows 5x faster inference on video tasks, allowing more scalable serving such as million-level video re-captioning. See instructions in our repo.**


key: _node_content
string_value:
{"id_": "890818b5-2c8f-420b-a1e4-73da3128a75f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26c13974-a5da-4e90-b232-3f28979c827f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e19f003-df73-4c34-924c-dfac3db8116a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "2e010215656dd890c8c51a164f91ea95373e33320f447f144b0c4683b1d328a3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3daa6017-6b85-49dc-b4ae-694096ff7d61", "node_type": "1", "metadata": {}, "hash": "f2c09721d494f545b6e99cce1c4744f9df6c23e6a2c7840dc5f61df1168ce519", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3386, "end_char_idx": 6352, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).
The original code can be found [here](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/inference).

## Usage tips

- We advise users to use `padding_side="left"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = "left"` before generating.

<Tip warning={true}>

- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is "left-padding" if model is in `eval()` mode, otherwise "right-padding".

</Tip>


- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use tokenizer's `apply_chat_template` to format your prompts correctly. Below is an example of how to do that.

We will use [LLaVA-NeXT-Video-7B-hf](https://huggingface.co/llava-hf/LLaVA-NeXT-Video-7B-hf) and a conversation history of videos and images. Each content field has to be a list of dicts, as follows:

```python
from transformers import LlavaNextVideoProcessor

processor = LlavaNextVideoProcessor.from_pretrained("llava-hf/LLaVA-NeXT-Video-7B-hf")

conversation = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions."},
            ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What’s shown in this image?"},
            {"type": "image"},
            ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Why is this video funny?"},
            {"type": "video"},
            ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your visuals
print(text_prompt)
```

## Usage example

### Single Media Mode

The model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):

```python
import av
import torch
import numpy as np
from transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor

def read_video_pyav(container, indices):
    '''
    Decode the video with PyAV decoder.
    Args:
        container (`av.container.input.InputContainer`): PyAV container.
        indices (`List[int]`): List of frame indices to decode.
    Returns:
        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
    '''


key: _node_content
string_value:
{"id_": "3daa6017-6b85-49dc-b4ae-694096ff7d61", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26c13974-a5da-4e90-b232-3f28979c827f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "890818b5-2c8f-420b-a1e4-73da3128a75f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "69d7cd7238273b8e666e9ed7db395dc2886620408fc64a374edc632e631b7461", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "665798f7-26dc-425d-abfa-d1040a97f264", "node_type": "1", "metadata": {}, "hash": "517c920fa1528ae0a23a983e7788761f55a82254c5980e254af75eaa75e746a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4870, "end_char_idx": 7859, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The assistant gives helpful, detailed, and polite answers to the human's questions."},
            ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What’s shown in this image?"},
            {"type": "image"},
            ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Why is this video funny?"},
            {"type": "video"},
            ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your visuals
print(text_prompt)
```

## Usage example

### Single Media Mode

The model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):

```python
import av
import torch
import numpy as np
from transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor

def read_video_pyav(container, indices):
    '''
    Decode the video with PyAV decoder.
    Args:
        container (`av.container.input.InputContainer`): PyAV container.
        indices (`List[int]`): List of frame indices to decode.
    Returns:
        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
    '''
    frames = []
    container.seek(0)
    start_index = indices[0]
    end_index = indices[-1]
    for i, frame in enumerate(container.decode(video=0)):
        if i > end_index:
            break
        if i >= start_index and i in indices:
            frames.append(frame)
    return np.stack([x.to_ndarray(format="rgb24") for x in frames])

# Load the model in half-precision
model = LlavaNextVideoForConditionalGeneration.from_pretrained("llava-hf/LLaVA-NeXT-Video-7B-hf", torch_dtype=torch.float16, device_map="auto")
processor = LlavaNextVideoProcessor.from_pretrained("llava-hf/LLaVA-NeXT-Video-7B-hf")

# Load the video as an np.array, sampling uniformly 8 frames (can sample more for longer videos)
video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
container = av.open(video_path)
total_frames = container.streams.video[0].frames
indices = np.arange(0, total_frames, total_frames / 8).astype(int)
video = read_video_pyav(container, indices)

conversation = [
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Why is this video funny?"},
            {"type": "video"},
            ],
    },
]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=prompt, videos=video, return_tensors="pt")

out = model.generate(**inputs, max_new_tokens=60)
processor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)
```


key: _node_content
string_value:
{"id_": "665798f7-26dc-425d-abfa-d1040a97f264", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26c13974-a5da-4e90-b232-3f28979c827f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3daa6017-6b85-49dc-b4ae-694096ff7d61", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "cef73082c08fdf10fc812fa6f050b4d307b6fe0dc6d7087756feaddd79e83442", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d9f2633-24b3-4403-ad65-1ebdb0998d35", "node_type": "1", "metadata": {}, "hash": "8e71dcaf6b64506d99358b9130c6a60b4d1923cf91d4323e535ff36c65a5ea36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6357, "end_char_idx": 9804, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
frames = []
    container.seek(0)
    start_index = indices[0]
    end_index = indices[-1]
    for i, frame in enumerate(container.decode(video=0)):
        if i > end_index:
            break
        if i >= start_index and i in indices:
            frames.append(frame)
    return np.stack([x.to_ndarray(format="rgb24") for x in frames])

# Load the model in half-precision
model = LlavaNextVideoForConditionalGeneration.from_pretrained("llava-hf/LLaVA-NeXT-Video-7B-hf", torch_dtype=torch.float16, device_map="auto")
processor = LlavaNextVideoProcessor.from_pretrained("llava-hf/LLaVA-NeXT-Video-7B-hf")

# Load the video as an np.array, sampling uniformly 8 frames (can sample more for longer videos)
video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
container = av.open(video_path)
total_frames = container.streams.video[0].frames
indices = np.arange(0, total_frames, total_frames / 8).astype(int)
video = read_video_pyav(container, indices)

conversation = [
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Why is this video funny?"},
            {"type": "video"},
            ],
    },
]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=prompt, videos=video, return_tensors="pt")

out = model.generate(**inputs, max_new_tokens=60)
processor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)
```


### Mixed Media Mode

The model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: 

```python
from PIL import Image
import requests

# Generate from image and video mixed inputs
# Load and image and write a new prompt
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
conversation = [
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "How many cats are there in the image?"},
            {"type": "image"},
            ],
    },
    {

        "role": "assistant",
        "content": [{"type": "text", "text": "There are two cats"}],
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Why is this video funny?"},
            {"type": "video"},
            ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors="pt")

# Generate
generate_ids = model.generate(**inputs, max_length=50)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)

```

## Model optimization

### Quantization using Bitsandbytes for memory efficiency

The model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. This allows for efficient deployment on resource-constrained cases. 

First make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:


key: _node_content
string_value:
{"id_": "4d9f2633-24b3-4403-ad65-1ebdb0998d35", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "26c13974-a5da-4e90-b232-3f28979c827f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "665798f7-26dc-425d-abfa-d1040a97f264", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers"}, "hash": "ef688eedae3f33c2a036c91591d4e2f8fb5710b72ec0fbc49a42dc6a09cd8d9c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9807, "end_char_idx": 11655, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor

# specify how to quantize the model
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = LlavaNextVideoForConditionalGeneration.from_pretrained("llava-hf/LLaVA-NeXT-Video-7B-hf", quantization_config=quantization_config, device_map="auto")
```


### Flash-Attention 2 to speed-up generation

Additionally, we can greatly speed-up model inference by using [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.

First, make sure to install the latest version of Flash Attention 2:

```bash
pip install -U flash-attn --no-build-isolation
```

Also, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.

To load and run a model using Flash Attention-2, simply add `attn_implementation="flash_attention_2"` when loading the model as follows:

```python
from transformers import LlavaNextVideoForConditionalGeneration

model = LlavaNextVideoForConditionalGeneration.from_pretrained(
    "llava-hf/LLaVA-NeXT-Video-7B-hf", 
    torch_dtype=torch.float16, 
    attn_implementation="flash_attention_2",
).to(0)
```



## LlavaNextVideoConfig

[[autodoc]] LlavaNextVideoConfig

## LlavaNextVideoProcessor

[[autodoc]] LlavaNextVideoProcessor

## LlavaNextVideoImageProcessor

[[autodoc]] LlavaNextVideoImageProcessor

## LlavaNextVideoForConditionalGeneration

[[autodoc]] LlavaNextVideoForConditionalGeneration
    - forward


key: _node_content
string_value:
{"id_": "2356badd-3798-4645-a2b9-826e388591a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2", "title": "XLSR-Wav2Vec2", "tokens": 728, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6b2749a9-7a61-44b2-a8d8-604c2506accb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2", "title": "XLSR-Wav2Vec2", "tokens": 728, "retrieve_doc": true, "source": "transformers"}, "hash": "346e0f3442ccd70629e1cdcee7c00d1ee4f4f70000b991c4f6c99b678b5498cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2269, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLSR-Wav2Vec2

## Overview

The XLSR-Wav2Vec2 model was proposed in [Unsupervised Cross-Lingual Representation Learning For Speech Recognition](https://arxiv.org/abs/2006.13979) by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael
Auli.

The abstract from the paper is the following:

*This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw
waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over
masked latent speech representations and jointly learns a quantization of the latents shared across languages. The
resulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly
outperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction
of 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to
a comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong
individual models. Analysis shows that the latent discrete speech representations are shared across languages with
increased sharing for related languages. We hope to catalyze research in low-resource speech understanding by releasing
XLSR-53, a large model pretrained in 53 languages.*

The original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).

Note: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).

## Usage tips

- XLSR-Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
- XLSR-Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be
  decoded using [`Wav2Vec2CTCTokenizer`].

<Tip>

XLSR-Wav2Vec2's architecture is based on the Wav2Vec2 model, so one can refer to [Wav2Vec2's documentation page](wav2vec2).

</Tip>


key: _node_content
string_value:
{"id_": "0e6fb29f-3da7-401c-b2d8-a36a47558c4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "hash": "887e9a14cc63016d22a7e940f7f04473dd0e16f4c3884228e28b2f6fc48e5524", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25814a6d-fd8a-45ea-b3e1-a74b8a425575", "node_type": "1", "metadata": {}, "hash": "7f3234812f55db0630f62818f2dfca439c0b58df9d761d6c19610cccc52b3107", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2883, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MVP

## Overview

The MVP model was proposed in [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://arxiv.org/abs/2206.12131) by Tianyi Tang, Junyi Li, Wayne Xin Zhao and Ji-Rong Wen.


According to the abstract,

- MVP follows a standard Transformer encoder-decoder architecture.
- MVP is supervised pre-trained using labeled datasets.
- MVP also has task-specific soft prompts to stimulate the model's capacity in performing a certain task.
- MVP is specially designed for natural language generation and can be adapted to a wide range of generation tasks, including but not limited to summarization, data-to-text generation, open-ended dialogue system, story generation, question answering, question generation, task-oriented dialogue system, commonsense generation, paraphrase generation, text style transfer, and text simplification. Our model can also be adapted to natural language understanding tasks such as sequence classification and (extractive) question answering.

This model was contributed by [Tianyi Tang](https://huggingface.co/StevenTang). The detailed information and instructions can be found [here](https://github.com/RUCAIBox/MVP).

## Usage tips

- We have released a series of models [here](https://huggingface.co/models?filter=mvp), including MVP, MVP with task-specific prompts, and multi-task pre-trained variants.
- If you want to use a model without prompts (standard Transformer), you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp')`.
- If you want to use a model with task-specific prompts, such as summarization, you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization')`.
- Our model supports lightweight prompt tuning following [Prefix-tuning](https://arxiv.org/abs/2101.00190) with method `set_lightweight_tuning()`.

## Usage examples

For summarization, it is an example to use MVP and MVP with summarization-specific prompts.

```python
>>> from transformers import MvpTokenizer, MvpForConditionalGeneration

>>> tokenizer = MvpTokenizer.from_pretrained("RUCAIBox/mvp")
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp-summarization")

>>> inputs = tokenizer(
...     "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.",
...     return_tensors="pt",
... )
>>> generated_ids = model.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Why You Shouldn't Quit Your Job"]

>>> generated_ids = model_with_prompt.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Don't do it if these are your reasons"]
```

For data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.


key: _node_content
string_value:
{"id_": "25814a6d-fd8a-45ea-b3e1-a74b8a425575", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "hash": "887e9a14cc63016d22a7e940f7f04473dd0e16f4c3884228e28b2f6fc48e5524", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e6fb29f-3da7-401c-b2d8-a36a47558c4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "hash": "db4697e91e0e69605a8488288c45eeb16242df0379eced24c631fb15d8412f4b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2630177d-e1d9-4995-aafa-d02f1efeb948", "node_type": "1", "metadata": {}, "hash": "4a633ea8106f4a3a4fbb4c31e5b159e1d3da56e1224fa656f6444362ce601d17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1188, "end_char_idx": 4079, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Usage tips

- We have released a series of models [here](https://huggingface.co/models?filter=mvp), including MVP, MVP with task-specific prompts, and multi-task pre-trained variants.
- If you want to use a model without prompts (standard Transformer), you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp')`.
- If you want to use a model with task-specific prompts, such as summarization, you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization')`.
- Our model supports lightweight prompt tuning following [Prefix-tuning](https://arxiv.org/abs/2101.00190) with method `set_lightweight_tuning()`.

## Usage examples

For summarization, it is an example to use MVP and MVP with summarization-specific prompts.

```python
>>> from transformers import MvpTokenizer, MvpForConditionalGeneration

>>> tokenizer = MvpTokenizer.from_pretrained("RUCAIBox/mvp")
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp-summarization")

>>> inputs = tokenizer(
...     "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.",
...     return_tensors="pt",
... )
>>> generated_ids = model.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Why You Shouldn't Quit Your Job"]

>>> generated_ids = model_with_prompt.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Don't do it if these are your reasons"]
```

For data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.
```python
>>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration

>>> tokenizer = MvpTokenizerFast.from_pretrained("RUCAIBox/mvp")
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mtl-data-to-text")

>>> inputs = tokenizer(
...     "Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man",
...     return_tensors="pt",
... )
>>> generated_ids = model.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']

>>> generated_ids = model_with_mtl.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']
```

For lightweight tuning, *i.e.*, fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the [original paper](https://arxiv.org/abs/2101.00190).


key: _node_content
string_value:
{"id_": "2630177d-e1d9-4995-aafa-d02f1efeb948", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "hash": "887e9a14cc63016d22a7e940f7f04473dd0e16f4c3884228e28b2f6fc48e5524", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25814a6d-fd8a-45ea-b3e1-a74b8a425575", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers"}, "hash": "24f95573d256242fd3fdd81eadb26ca0280f2ebef7c52e1a4c444bc88773d22e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2430, "end_char_idx": 5837, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     return_tensors="pt",
... )
>>> generated_ids = model.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Why You Shouldn't Quit Your Job"]

>>> generated_ids = model_with_prompt.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
["Don't do it if these are your reasons"]
```

For data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.
```python
>>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration

>>> tokenizer = MvpTokenizerFast.from_pretrained("RUCAIBox/mvp")
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp")
>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mtl-data-to-text")

>>> inputs = tokenizer(
...     "Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man",
...     return_tensors="pt",
... )
>>> generated_ids = model.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']

>>> generated_ids = model_with_mtl.generate(**inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']
```

For lightweight tuning, *i.e.*, fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the [original paper](https://arxiv.org/abs/2101.00190).

```python
>>> from transformers import MvpForConditionalGeneration

>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp", use_prompt=True)
>>> # the number of trainable parameters (full tuning)
>>> sum(p.numel() for p in model.parameters() if p.requires_grad)
468116832

>>> # lightweight tuning with randomly initialized prompts
>>> model.set_lightweight_tuning()
>>> # the number of trainable parameters (lightweight tuning)
>>> sum(p.numel() for p in model.parameters() if p.requires_grad)
61823328

>>> # lightweight tuning with task-specific prompts
>>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mtl-data-to-text")
>>> model.set_lightweight_tuning()
>>> # original lightweight Prefix-tuning
>>> model = MvpForConditionalGeneration.from_pretrained("facebook/bart-large", use_prompt=True)
>>> model.set_lightweight_tuning()
```

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## MvpConfig

[[autodoc]] MvpConfig

## MvpTokenizer

[[autodoc]] MvpTokenizer

## MvpTokenizerFast

[[autodoc]] MvpTokenizerFast

## MvpModel

[[autodoc]] MvpModel
    - forward

## MvpForConditionalGeneration

[[autodoc]] MvpForConditionalGeneration
    - forward

## MvpForSequenceClassification

[[autodoc]] MvpForSequenceClassification
    - forward

## MvpForQuestionAnswering

[[autodoc]] MvpForQuestionAnswering
    - forward

## MvpForCausalLM

[[autodoc]] MvpForCausalLM
    - forward


key: _node_content
string_value:
{"id_": "1164353e-1a01-455d-a18b-65ff1f5c337d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68484f21-0320-45fc-85fe-ce27fdafdf7d", "node_type": "1", "metadata": {}, "hash": "8f2925fd6c46cee35196f44f97ef807ca7309c72b612a73365cb45d2fb96a125", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2768, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MBart and MBart-50

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=mbart">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-mbart-blueviolet">
</a>
<a href="https://huggingface.co/spaces/docs-demos/mbart-large-50-one-to-many-mmt">
<img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
</a>
</div>


## Overview of MBart

The MBart model was presented in [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan
Ghazvininejad, Mike Lewis, Luke Zettlemoyer.

According to the abstract, MBART is a sequence-to-sequence denoising auto-encoder pretrained on large-scale monolingual
corpora in many languages using the BART objective. mBART is one of the first methods for pretraining a complete
sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only
on the encoder, decoder, or reconstructing parts of the text.

This model was contributed by [valhalla](https://huggingface.co/valhalla). The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/mbart)

### Training of MBart

MBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for translation task. As the
model is multilingual it expects the sequences in a different format. A special language id token is added in both the
source and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The
target text format is `[tgt_lang_code] X [eos]`. `bos` is never used.

The regular [`~MBartTokenizer.__call__`] will encode source text format passed as first argument or with the `text`
keyword, and target text format passed with the `text_label` keyword argument.

- Supervised training

```python
>>> from transformers import MBartForConditionalGeneration, MBartTokenizer

>>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO")
>>> example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
>>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria"

>>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt")

>>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-en-ro")
>>> # forward pass
>>> model(**inputs)
```

- Generation

  While generating the target text set the `decoder_start_token_id` to the target language id. The following
  example shows how to translate English to Romanian using the *facebook/mbart-large-en-ro* model.


key: _node_content
string_value:
{"id_": "68484f21-0320-45fc-85fe-ce27fdafdf7d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1164353e-1a01-455d-a18b-65ff1f5c337d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "50f4ccd87ca844698bc22bc31462b378be7885029fe5763501b4f8f26590a167", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3adfdae5-184d-4855-9a84-f3f57773c1e8", "node_type": "1", "metadata": {}, "hash": "7d6a9301c89809ed9f5de04da89188082c482b01652de5c58c0a0388581f63ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1162, "end_char_idx": 4380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/mbart)

### Training of MBart

MBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for translation task. As the
model is multilingual it expects the sequences in a different format. A special language id token is added in both the
source and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The
target text format is `[tgt_lang_code] X [eos]`. `bos` is never used.

The regular [`~MBartTokenizer.__call__`] will encode source text format passed as first argument or with the `text`
keyword, and target text format passed with the `text_label` keyword argument.

- Supervised training

```python
>>> from transformers import MBartForConditionalGeneration, MBartTokenizer

>>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO")
>>> example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
>>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria"

>>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt")

>>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-en-ro")
>>> # forward pass
>>> model(**inputs)
```

- Generation

  While generating the target text set the `decoder_start_token_id` to the target language id. The following
  example shows how to translate English to Romanian using the *facebook/mbart-large-en-ro* model.

```python
>>> from transformers import MBartForConditionalGeneration, MBartTokenizer

>>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX")
>>> article = "UN Chief Says There Is No Military Solution in Syria"
>>> inputs = tokenizer(article, return_tensors="pt")
>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id["ro_RO"])
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"Şeful ONU declară că nu există o soluţie militară în Siria"
```

## Overview of MBart-50

MBart-50 was introduced in the [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) paper by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav
Chaudhary, Jiatao Gu, Angela Fan. MBart-50 is created using the original *mbart-large-cc25* checkpoint by extendeding
its embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50
languages.

According to the abstract

*Multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one
direction, a pretrained model is finetuned on many directions at the same time. It demonstrates that pretrained models
can be extended to incorporate additional languages without loss of performance. Multilingual finetuning improves on
average 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while
improving 9.3 BLEU on average over bilingual baselines from scratch.*


key: _node_content
string_value:
{"id_": "3adfdae5-184d-4855-9a84-f3f57773c1e8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68484f21-0320-45fc-85fe-ce27fdafdf7d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "94cc98f07015416a55971aa31049e87578efa48682ef655b64290776f43e4065", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8621d0d7-54f6-4f03-9632-9744942b0731", "node_type": "1", "metadata": {}, "hash": "1c7ed3d03c5d33f8f421f6e8976baf8920d60ea504d690d142b74a05b6920824", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2770, "end_char_idx": 5870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> from transformers import MBartForConditionalGeneration, MBartTokenizer

>>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX")
>>> article = "UN Chief Says There Is No Military Solution in Syria"
>>> inputs = tokenizer(article, return_tensors="pt")
>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id["ro_RO"])
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
"Şeful ONU declară că nu există o soluţie militară în Siria"
```

## Overview of MBart-50

MBart-50 was introduced in the [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) paper by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav
Chaudhary, Jiatao Gu, Angela Fan. MBart-50 is created using the original *mbart-large-cc25* checkpoint by extendeding
its embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50
languages.

According to the abstract

*Multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one
direction, a pretrained model is finetuned on many directions at the same time. It demonstrates that pretrained models
can be extended to incorporate additional languages without loss of performance. Multilingual finetuning improves on
average 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while
improving 9.3 BLEU on average over bilingual baselines from scratch.*


### Training of MBart-50

The text format for MBart-50 is slightly different from mBART. For MBart-50 the language id token is used as a prefix
for both source and target text i.e the text format is `[lang_code] X [eos]`, where `lang_code` is source
language id for source text and target language id for target text, with `X` being the source or target text
respectively.


MBart-50 has its own tokenizer [`MBart50Tokenizer`].

-  Supervised training

```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50", src_lang="en_XX", tgt_lang="ro_RO")

src_text = " UN Chief Says There Is No Military Solution in Syria"
tgt_text = "Şeful ONU declară că nu există o soluţie militară în Siria"

model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")

model(**model_inputs)  # forward pass
```

- Generation

  To generate using the mBART-50 multilingual translation models, `eos_token_id` is used as the
  `decoder_start_token_id` and the target language id is forced as the first generated token. To force the
  target language id as the first generated token, pass the *forced_bos_token_id* parameter to the *generate* method.
  The following example shows how to translate between Hindi to French and Arabic to English using the
  *facebook/mbart-50-large-many-to-many* checkpoint.


key: _node_content
string_value:
{"id_": "8621d0d7-54f6-4f03-9632-9744942b0731", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3adfdae5-184d-4855-9a84-f3f57773c1e8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "bb048c9822ed7baa387f47ccaf22bf81238dabce26f1eb13eaffd7cc70d53d4b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c8ffc65-b65e-4825-8e17-4665ee5d4ff9", "node_type": "1", "metadata": {}, "hash": "4346abfd61c417c7387c8b4b974d86dcdcb2d0fe3045e27dc26fb974396f5007", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4379, "end_char_idx": 7071, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*


### Training of MBart-50

The text format for MBart-50 is slightly different from mBART. For MBart-50 the language id token is used as a prefix
for both source and target text i.e the text format is `[lang_code] X [eos]`, where `lang_code` is source
language id for source text and target language id for target text, with `X` being the source or target text
respectively.


MBart-50 has its own tokenizer [`MBart50Tokenizer`].

-  Supervised training

```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50", src_lang="en_XX", tgt_lang="ro_RO")

src_text = " UN Chief Says There Is No Military Solution in Syria"
tgt_text = "Şeful ONU declară că nu există o soluţie militară în Siria"

model_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors="pt")

model(**model_inputs)  # forward pass
```

- Generation

  To generate using the mBART-50 multilingual translation models, `eos_token_id` is used as the
  `decoder_start_token_id` and the target language id is forced as the first generated token. To force the
  target language id as the first generated token, pass the *forced_bos_token_id* parameter to the *generate* method.
  The following example shows how to translate between Hindi to French and Arabic to English using the
  *facebook/mbart-50-large-many-to-many* checkpoint.

```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"
article_ar = "الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"
encoded_hi = tokenizer(article_hi, return_tensors="pt")
generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria."

# translate Arabic to English
tokenizer.src_lang = "ar_AR"
encoded_ar = tokenizer(article_ar, return_tensors="pt")
generated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "The Secretary-General of the United Nations says there is no military solution in Syria."


key: _node_content
string_value:
{"id_": "2c8ffc65-b65e-4825-8e17-4665ee5d4ff9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8621d0d7-54f6-4f03-9632-9744942b0731", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers"}, "hash": "013f4ccfd221f051de2036473e0a25d792dbc98469991927a6f247b3dffd2692", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5717, "end_char_idx": 8786, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The following example shows how to translate between Hindi to French and Arabic to English using the
  *facebook/mbart-50-large-many-to-many* checkpoint.

```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"
article_ar = "الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"
encoded_hi = tokenizer(article_hi, return_tensors="pt")
generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria."

# translate Arabic to English
tokenizer.src_lang = "ar_AR"
encoded_ar = tokenizer(article_ar, return_tensors="pt")
generated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "The Secretary-General of the United Nations says there is no military solution in Syria."
```

## Documentation resources

- [Text classification task guide](../tasks/sequence_classification)
- [Question answering task guide](../tasks/question_answering)
- [Causal language modeling task guide](../tasks/language_modeling)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## MBartConfig

[[autodoc]] MBartConfig

## MBartTokenizer

[[autodoc]] MBartTokenizer
    - build_inputs_with_special_tokens

## MBartTokenizerFast

[[autodoc]] MBartTokenizerFast

## MBart50Tokenizer

[[autodoc]] MBart50Tokenizer

## MBart50TokenizerFast

[[autodoc]] MBart50TokenizerFast

<frameworkcontent>
<pt>

## MBartModel

[[autodoc]] MBartModel

## MBartForConditionalGeneration

[[autodoc]] MBartForConditionalGeneration

## MBartForQuestionAnswering

[[autodoc]] MBartForQuestionAnswering

## MBartForSequenceClassification

[[autodoc]] MBartForSequenceClassification

## MBartForCausalLM

[[autodoc]] MBartForCausalLM
    - forward

</pt>
<tf>

## TFMBartModel

[[autodoc]] TFMBartModel
    - call

## TFMBartForConditionalGeneration

[[autodoc]] TFMBartForConditionalGeneration
    - call

</tf>
<jax>

## FlaxMBartModel

[[autodoc]] FlaxMBartModel
    - __call__
    - encode
    - decode

## FlaxMBartForConditionalGeneration

[[autodoc]] FlaxMBartForConditionalGeneration
    - __call__
    - encode
    - decode

## FlaxMBartForSequenceClassification

[[autodoc]] FlaxMBartForSequenceClassification
    - __call__
    - encode
    - decode

## FlaxMBartForQuestionAnswering

[[autodoc]] FlaxMBartForQuestionAnswering
    - __call__
    - encode
    - decode

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "18dec7ae-5709-422c-881f-8dd749d4c822", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2", "node_type": "1", "metadata": {}, "hash": "4e22a39c16710616f3524a09afb18e0365404d42c4aca02bd883bc532e01407d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3427, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Whisper

## Overview

The Whisper model was proposed in [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf) by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.

The abstract from the paper is the following:

*We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.*

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).
The original code can be found [here](https://github.com/openai/whisper).

## Usage tips

- The model usually performs well without requiring any finetuning.
- The architecture follows a classic encoder-decoder architecture, which means that it relies on the [`~generation.GenerationMixin.generate`] function for inference.
- One can use [`WhisperProcessor`] to prepare audio for the model, and decode the predicted ID's back into text.

- To convert the model and the processor, we recommend using the following:

```bash
python src/transformers/models/whisper/convert_openai_to_hf.py --checkpoint_path "" --pytorch_dump_folder_path "Arthur/whisper-3" --convert_preprocessor True
```
The script will automatically determine all necessary parameters from the OpenAI checkpoint. A `tiktoken` library needs to be installed
to perform the conversion of the OpenAI tokenizer to the `tokenizers` version.

## Inference

Here is a step-by-step guide to transcribing an audio sample using a pre-trained Whisper model:

```python
>>> from datasets import load_dataset
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration

>>> # Select an audio file and read it:
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> audio_sample = ds[0]["audio"]

>>> # Load the Whisper model in Hugging Face format:
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")

>>> # Use the model and processor to transcribe the audio:
>>> input_features = processor(
...     audio_sample["array"], sampling_rate=audio_sample["sampling_rate"], return_tensors="pt"
... ).input_features

>>> # Generate token ids
>>> predicted_ids = model.generate(input_features)

>>> # Decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

>>> transcription[0]
' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'
```

Whisper is compatible with the following optimisations for both short and long-form generation:
- [PyTorch Scaled Dot Product Attention (SDPA)](../perf_infer_gpu_one#pytorch-scaled-dot-product-attention): flash attention and memory-efficient attention kernels. Enabled by default for `torch>=2.1.1`.


key: _node_content
string_value:
{"id_": "3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "18dec7ae-5709-422c-881f-8dd749d4c822", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "88df6a4bd1ddbed60f11986fa864845c71ea287785eb96d75000003c930bb723", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47eba35f-7b05-45bf-a8c0-f797ff1090fd", "node_type": "1", "metadata": {}, "hash": "d9d8d4c8c3ab09ee309bc50ffd061ccad1082ffc59b7c0e42be5607ee963f5c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1860, "end_char_idx": 5293, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A `tiktoken` library needs to be installed
to perform the conversion of the OpenAI tokenizer to the `tokenizers` version.

## Inference

Here is a step-by-step guide to transcribing an audio sample using a pre-trained Whisper model:

```python
>>> from datasets import load_dataset
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration

>>> # Select an audio file and read it:
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> audio_sample = ds[0]["audio"]

>>> # Load the Whisper model in Hugging Face format:
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")

>>> # Use the model and processor to transcribe the audio:
>>> input_features = processor(
...     audio_sample["array"], sampling_rate=audio_sample["sampling_rate"], return_tensors="pt"
... ).input_features

>>> # Generate token ids
>>> predicted_ids = model.generate(input_features)

>>> # Decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

>>> transcription[0]
' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'
```

Whisper is compatible with the following optimisations for both short and long-form generation:
- [PyTorch Scaled Dot Product Attention (SDPA)](../perf_infer_gpu_one#pytorch-scaled-dot-product-attention): flash attention and memory-efficient attention kernels. Enabled by default for `torch>=2.1.1`.
- [Flash Attention 2](../perf_infer_gpu_one#flashattention-2): improved implementation of flash attention through better parallelism and work partitioning. 
- [torch.compile](../llm_optims#static-kv-cache-and-torchcompile): JIT-compile the forward pass to dispatch to efficient fused kernels.

As an example, the following codesnippet enables SDPA and `torch.compile` for up to 5x faster inference:

```python
>>> from datasets import load_dataset
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration

>>> # Select an audio file and read it:
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> audio_sample = ds[0]["audio"]

>>> # Load the Whisper model with SDPA attention
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", attn_implementation="sdpa")

>>> # Enable static cache and compile the forward pass
>>> model.generation_config.cache_implementation = "static"
>>> model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)

>>> # Use the model and processor to transcribe the audio:
>>> input_features = processor(
...     audio_sample["array"], sampling_rate=audio_sample["sampling_rate"], return_tensors="pt"
... ).input_features

>>> # Compile the forward pass
>>> for _ in range(2):
>>>     model.generate(input_features)

>>> # Generate token ids using compiled graph (fast!)
>>> predicted_ids = model.generate(input_features)

>>> # Decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

>>> transcription[0]
' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'
```

For more details on each optimisation, refer to the documentation linked above.


key: _node_content
string_value:
{"id_": "47eba35f-7b05-45bf-a8c0-f797ff1090fd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "a532d81fb42603da715df8dc765c077bc45f4b7826f9a7b24fbf62613d41ceb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf2bc405-210d-4b01-b1a2-534d87d2c2d3", "node_type": "1", "metadata": {}, "hash": "92efc6becf9843ea07cb93a5f1260f8908068d112a667e6e2178ddbf707b2ed6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3585, "end_char_idx": 6262, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- [torch.compile](../llm_optims#static-kv-cache-and-torchcompile): JIT-compile the forward pass to dispatch to efficient fused kernels.

As an example, the following codesnippet enables SDPA and `torch.compile` for up to 5x faster inference:

```python
>>> from datasets import load_dataset
>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration

>>> # Select an audio file and read it:
>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> audio_sample = ds[0]["audio"]

>>> # Load the Whisper model with SDPA attention
>>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
>>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", attn_implementation="sdpa")

>>> # Enable static cache and compile the forward pass
>>> model.generation_config.cache_implementation = "static"
>>> model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)

>>> # Use the model and processor to transcribe the audio:
>>> input_features = processor(
...     audio_sample["array"], sampling_rate=audio_sample["sampling_rate"], return_tensors="pt"
... ).input_features

>>> # Compile the forward pass
>>> for _ in range(2):
>>>     model.generate(input_features)

>>> # Generate token ids using compiled graph (fast!)
>>> predicted_ids = model.generate(input_features)

>>> # Decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

>>> transcription[0]
' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'
```

For more details on each optimisation, refer to the documentation linked above.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Whisper. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- [Fine-tune Whisper](https://huggingface.co/blog/fine-tune-whisper) on your own dataset for better downstream performance.
- [Distil-Whisper](https://huggingface.co/distil-whisper): Upto 6x faster, 2x smaller distilled Whisper models for English. We release the [model checkpoints](https://huggingface.co/distil-whisper), and [distillation code](https://github.com/huggingface/distil-whisper).
- A fork with a script to [convert a Whisper model in Hugging Face format to OpenAI format](https://github.com/zuazo-forks/transformers/blob/convert_hf_to_openai/src/transformers/models/whisper/convert_hf_to_openai.py).


key: _node_content
string_value:
{"id_": "cf2bc405-210d-4b01-b1a2-534d87d2c2d3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47eba35f-7b05-45bf-a8c0-f797ff1090fd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers"}, "hash": "96d0c8a8e2f1ce2f0db7a524fca69d17a8c4ed35154c4e7434e4e4038b37f914", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4928, "end_char_idx": 8129, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> predicted_ids = model.generate(input_features)

>>> # Decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

>>> transcription[0]
' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'
```

For more details on each optimisation, refer to the documentation linked above.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Whisper. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

- [Fine-tune Whisper](https://huggingface.co/blog/fine-tune-whisper) on your own dataset for better downstream performance.
- [Distil-Whisper](https://huggingface.co/distil-whisper): Upto 6x faster, 2x smaller distilled Whisper models for English. We release the [model checkpoints](https://huggingface.co/distil-whisper), and [distillation code](https://github.com/huggingface/distil-whisper).
- A fork with a script to [convert a Whisper model in Hugging Face format to OpenAI format](https://github.com/zuazo-forks/transformers/blob/convert_hf_to_openai/src/transformers/models/whisper/convert_hf_to_openai.py). 🌎
Usage example:
```bash
pip install -U openai-whisper
python convert_hf_to_openai.py \
    --checkpoint openai/whisper-tiny \
    --whisper_dump_path whisper-tiny-openai.pt
```

## WhisperConfig

[[autodoc]] WhisperConfig

## WhisperTokenizer

[[autodoc]] WhisperTokenizer
    - set_prefix_tokens
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary
    - batch_decode
    - decode
    - basic_normalize
    - normalize

## WhisperTokenizerFast

[[autodoc]] WhisperTokenizerFast
    - set_prefix_tokens
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary
    - batch_decode
    - decode
    - basic_normalize
    - normalize

## WhisperFeatureExtractor

[[autodoc]] WhisperFeatureExtractor
    - __call__

## WhisperProcessor

[[autodoc]] WhisperProcessor
    - __call__
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

<frameworkcontent>
<pt>

## WhisperModel

[[autodoc]] WhisperModel
    - forward
    - _mask_input_features

## WhisperForConditionalGeneration

[[autodoc]] WhisperForConditionalGeneration
    - forward
    - generate

## WhisperForCausalLM

[[autodoc]] WhisperForCausalLM
    - forward

## WhisperForAudioClassification

[[autodoc]] WhisperForAudioClassification
    - forward

</pt>
<tf>

## TFWhisperModel

[[autodoc]] TFWhisperModel
    - call

## TFWhisperForConditionalGeneration

[[autodoc]] TFWhisperForConditionalGeneration
    - call

</tf>
<jax>

## FlaxWhisperModel

[[autodoc]] FlaxWhisperModel
    - __call__

## FlaxWhisperForConditionalGeneration

[[autodoc]] FlaxWhisperForConditionalGeneration
    - __call__

## FlaxWhisperForAudioClassification

[[autodoc]] FlaxWhisperForAudioClassification
    - __call__

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "b8787b14-d2a3-4b00-8e1a-e425fe67dab2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0542091b-e628-4e97-af39-64766bddac08", "node_type": "1", "metadata": {}, "hash": "dffe5e2e6b910278b431176684075a98e64351ae11c1fbe3912ec5934959ba0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LayoutLMv3

## Overview

The LayoutLMv3 model was proposed in [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://arxiv.org/abs/2204.08387) by Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei.
LayoutLMv3 simplifies [LayoutLMv2](layoutlmv2) by using patch embeddings (as in [ViT](vit)) instead of leveraging a CNN backbone, and pre-trains the model on 3 objectives: masked language modeling (MLM), masked image modeling (MIM)
and word-patch alignment (WPA).

The abstract from the paper is the following:

*Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/layoutlmv3_architecture.png"
alt="drawing" width="600"/>

<small> LayoutLMv3 architecture. Taken from the <a href="https://arxiv.org/abs/2204.08387">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [chriskoo](https://huggingface.co/chriskoo), [tokec](https://huggingface.co/tokec), and [lre](https://huggingface.co/lre). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/layoutlmv3).

## Usage tips

- In terms of data processing, LayoutLMv3 is identical to its predecessor [LayoutLMv2](layoutlmv2), except that:
    - images need to be resized and normalized with channels in regular RGB format. LayoutLMv2 on the other hand normalizes the images internally and expects the channels in BGR format.
    - text is tokenized using byte-pair encoding (BPE), as opposed to WordPiece.
  Due to these differences in data preprocessing, one can use [`LayoutLMv3Processor`] which internally combines a [`LayoutLMv3ImageProcessor`] (for the image modality) and a [`LayoutLMv3Tokenizer`]/[`LayoutLMv3TokenizerFast`] (for the text modality) to prepare all data for the model.
- Regarding usage of [`LayoutLMv3Processor`], we refer to the [usage guide](layoutlmv2#usage-layoutlmv2processor) of its predecessor.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLMv3.


key: _node_content
string_value:
{"id_": "0542091b-e628-4e97-af39-64766bddac08", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b8787b14-d2a3-4b00-8e1a-e425fe67dab2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "57ae3fdee1c5cb120beb86bd50e8bee629506860197026d05aa2148fcee8e87d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f37e59f9-7588-4deb-bca1-093508856f86", "node_type": "1", "metadata": {}, "hash": "1aae3006127329e62b24f53eafb906414bce3f9146f9bacb111d67e0a09c6031", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1882, "end_char_idx": 4747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the <a href="https://arxiv.org/abs/2204.08387">original paper</a>. </small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [chriskoo](https://huggingface.co/chriskoo), [tokec](https://huggingface.co/tokec), and [lre](https://huggingface.co/lre). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/layoutlmv3).

## Usage tips

- In terms of data processing, LayoutLMv3 is identical to its predecessor [LayoutLMv2](layoutlmv2), except that:
    - images need to be resized and normalized with channels in regular RGB format. LayoutLMv2 on the other hand normalizes the images internally and expects the channels in BGR format.
    - text is tokenized using byte-pair encoding (BPE), as opposed to WordPiece.
  Due to these differences in data preprocessing, one can use [`LayoutLMv3Processor`] which internally combines a [`LayoutLMv3ImageProcessor`] (for the image modality) and a [`LayoutLMv3Tokenizer`]/[`LayoutLMv3TokenizerFast`] (for the text modality) to prepare all data for the model.
- Regarding usage of [`LayoutLMv3Processor`], we refer to the [usage guide](layoutlmv2#usage-layoutlmv2processor) of its predecessor.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLMv3. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<Tip>

LayoutLMv3 is nearly identical to LayoutLMv2, so we've also included LayoutLMv2 resources you can adapt for LayoutLMv3 tasks. For these notebooks, take care to use [`LayoutLMv2Processor`] instead when preparing data for the model!

</Tip>

- Demo notebooks for LayoutLMv3 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3).
- Demo scripts can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3).

<PipelineTag pipeline="text-classification"/>

- [`LayoutLMv2ForSequenceClassification`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`LayoutLMv3ForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3) and [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb).


key: _node_content
string_value:
{"id_": "f37e59f9-7588-4deb-bca1-093508856f86", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0542091b-e628-4e97-af39-64766bddac08", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "0118d6e526145e53f7bd7e40615ae7e405aa4ef2d175b8807d35c80341fe39df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7c772c3-9430-4ff7-8476-167920c2af46", "node_type": "1", "metadata": {}, "hash": "2488f3cc1b0f99cdce812989fb0d66b521eebe10dd9ced5cab542bbd25139b0c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3128, "end_char_idx": 5922, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with LayoutLMv3. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<Tip>

LayoutLMv3 is nearly identical to LayoutLMv2, so we've also included LayoutLMv2 resources you can adapt for LayoutLMv3 tasks. For these notebooks, take care to use [`LayoutLMv2Processor`] instead when preparing data for the model!

</Tip>

- Demo notebooks for LayoutLMv3 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3).
- Demo scripts can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3).

<PipelineTag pipeline="text-classification"/>

- [`LayoutLMv2ForSequenceClassification`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).
- [Text classification task guide](../tasks/sequence_classification)

<PipelineTag pipeline="token-classification"/>

- [`LayoutLMv3ForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3) and [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb).
- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb) for how to perform inference with [`LayoutLMv2ForTokenClassification`] and a [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/True_inference_with_LayoutLMv2ForTokenClassification_%2B_Gradio_demo.ipynb) for how to perform inference when no labels are available with [`LayoutLMv2ForTokenClassification`].
- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) for how to finetune [`LayoutLMv2ForTokenClassification`] with the 🤗 Trainer.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="question-answering"/>

- [`LayoutLMv2ForQuestionAnswering`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).


key: _node_content
string_value:
{"id_": "c7c772c3-9430-4ff7-8476-167920c2af46", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f37e59f9-7588-4deb-bca1-093508856f86", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers"}, "hash": "27ae6aa3482fe78435c1859b23b62486c0b181e0d45c456d33347243b608840c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4748, "end_char_idx": 7346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb) for how to perform inference with [`LayoutLMv2ForTokenClassification`] and a [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/True_inference_with_LayoutLMv2ForTokenClassification_%2B_Gradio_demo.ipynb) for how to perform inference when no labels are available with [`LayoutLMv2ForTokenClassification`].
- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) for how to finetune [`LayoutLMv2ForTokenClassification`] with the 🤗 Trainer.
- [Token classification task guide](../tasks/token_classification)

<PipelineTag pipeline="question-answering"/>

- [`LayoutLMv2ForQuestionAnswering`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).
- [Question answering task guide](../tasks/question_answering)

**Document question answering**
- [Document question answering task guide](../tasks/document_question_answering)

## LayoutLMv3Config

[[autodoc]] LayoutLMv3Config

## LayoutLMv3FeatureExtractor

[[autodoc]] LayoutLMv3FeatureExtractor
    - __call__

## LayoutLMv3ImageProcessor

[[autodoc]] LayoutLMv3ImageProcessor
    - preprocess

## LayoutLMv3Tokenizer

[[autodoc]] LayoutLMv3Tokenizer
    - __call__
    - save_vocabulary

## LayoutLMv3TokenizerFast

[[autodoc]] LayoutLMv3TokenizerFast
    - __call__

## LayoutLMv3Processor

[[autodoc]] LayoutLMv3Processor
    - __call__

<frameworkcontent>
<pt>

## LayoutLMv3Model

[[autodoc]] LayoutLMv3Model
    - forward

## LayoutLMv3ForSequenceClassification

[[autodoc]] LayoutLMv3ForSequenceClassification
    - forward

## LayoutLMv3ForTokenClassification

[[autodoc]] LayoutLMv3ForTokenClassification
    - forward

## LayoutLMv3ForQuestionAnswering

[[autodoc]] LayoutLMv3ForQuestionAnswering
    - forward

</pt>
<tf>

## TFLayoutLMv3Model

[[autodoc]] TFLayoutLMv3Model
    - call

## TFLayoutLMv3ForSequenceClassification

[[autodoc]] TFLayoutLMv3ForSequenceClassification
    - call

## TFLayoutLMv3ForTokenClassification

[[autodoc]] TFLayoutLMv3ForTokenClassification
    - call

## TFLayoutLMv3ForQuestionAnswering

[[autodoc]] TFLayoutLMv3ForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "dbe94015-d003-40bc-978b-3afb357f10a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deformable_detr", "title": "Deformable DETR", "tokens": 934, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "94466d58-1b0e-443d-aca7-6ebef02b4837", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deformable_detr", "title": "Deformable DETR", "tokens": 934, "retrieve_doc": true, "source": "transformers"}, "hash": "856b34e792293fc7baec61f11086da2e16c48653b31dd25f3ab931745405e6f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3267, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Deformable DETR

## Overview

The Deformable DETR model was proposed in [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159) by Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai.
Deformable DETR mitigates the slow convergence issues and limited feature spatial resolution of the original [DETR](detr) by leveraging a new deformable attention module which only attends to a small set of key sampling points around a reference.

The abstract from the paper is the following:

*DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10 times less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/deformable_detr_architecture.png"
alt="drawing" width="600"/>

<small> Deformable DETR architecture. Taken from the <a href="https://arxiv.org/abs/2010.04159">original paper</a>.</small>

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/fundamentalvision/Deformable-DETR).

## Usage tips

- Training Deformable DETR is equivalent to training the original [DETR](detr) model. See the [resources](#resources) section below for demo notebooks.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with Deformable DETR.

<PipelineTag pipeline="object-detection"/>

- Demo notebooks regarding inference + fine-tuning on a custom dataset for [`DeformableDetrForObjectDetection`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Deformable-DETR).
- Scripts for finetuning [`DeformableDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).
- See also: [Object detection task guide](../tasks/object_detection).

If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## DeformableDetrImageProcessor

[[autodoc]] DeformableDetrImageProcessor
    - preprocess
    - post_process_object_detection

## DeformableDetrFeatureExtractor

[[autodoc]] DeformableDetrFeatureExtractor
    - __call__
    - post_process_object_detection

## DeformableDetrConfig

[[autodoc]] DeformableDetrConfig

## DeformableDetrModel

[[autodoc]] DeformableDetrModel
    - forward

## DeformableDetrForObjectDetection

[[autodoc]] DeformableDetrForObjectDetection
    - forward


key: _node_content
string_value:
{"id_": "6e6a404a-8801-4521-a2ac-b87a5eebae2a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "2186a2a04b5fb3a1ecdbb8a9d962233c361326beb388a7dbd6c242657ae92abd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d1c85d7-05ff-4337-8b6f-123659dc4663", "node_type": "1", "metadata": {}, "hash": "7061fb6d69c3beec448cbbb28a29e6878f2b3e380e1f2312c4f5a95746853acf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2492, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BLIP

## Overview

The BLIP model was proposed in [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086) by Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi.

BLIP is a model that is able to perform various multi-modal tasks including:
- Visual Question Answering 
- Image-Text retrieval (Image-text matching)
- Image Captioning

The abstract from the paper is the following:

*Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. 
However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*

![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif)

This model was contributed by [ybelkada](https://huggingface.co/ybelkada).
The original code can be found [here](https://github.com/salesforce/BLIP).

## Resources

- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset

## BlipConfig

[[autodoc]] BlipConfig
    - from_text_vision_configs

## BlipTextConfig

[[autodoc]] BlipTextConfig

## BlipVisionConfig

[[autodoc]] BlipVisionConfig

## BlipProcessor

[[autodoc]] BlipProcessor

## BlipImageProcessor

[[autodoc]] BlipImageProcessor
    - preprocess

<frameworkcontent>
<pt>

## BlipModel

`BlipModel` is going to be deprecated in future versions, please use `BlipForConditionalGeneration`, `BlipForImageTextRetrieval` or `BlipForQuestionAnswering` depending on your usecase.


key: _node_content
string_value:
{"id_": "1d1c85d7-05ff-4337-8b6f-123659dc4663", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "2186a2a04b5fb3a1ecdbb8a9d962233c361326beb388a7dbd6c242657ae92abd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e6a404a-8801-4521-a2ac-b87a5eebae2a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers"}, "hash": "f1eb5a8e936326b5f5a67e8eff4e5251eba32d7535e1c6853bcc92a3023be554", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1161, "end_char_idx": 3472, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*

![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif)

This model was contributed by [ybelkada](https://huggingface.co/ybelkada).
The original code can be found [here](https://github.com/salesforce/BLIP).

## Resources

- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset

## BlipConfig

[[autodoc]] BlipConfig
    - from_text_vision_configs

## BlipTextConfig

[[autodoc]] BlipTextConfig

## BlipVisionConfig

[[autodoc]] BlipVisionConfig

## BlipProcessor

[[autodoc]] BlipProcessor

## BlipImageProcessor

[[autodoc]] BlipImageProcessor
    - preprocess

<frameworkcontent>
<pt>

## BlipModel

`BlipModel` is going to be deprecated in future versions, please use `BlipForConditionalGeneration`, `BlipForImageTextRetrieval` or `BlipForQuestionAnswering` depending on your usecase.

[[autodoc]] BlipModel
    - forward
    - get_text_features
    - get_image_features

## BlipTextModel

[[autodoc]] BlipTextModel
    - forward

## BlipVisionModel

[[autodoc]] BlipVisionModel
    - forward

## BlipForConditionalGeneration

[[autodoc]] BlipForConditionalGeneration
    - forward

## BlipForImageTextRetrieval

[[autodoc]] BlipForImageTextRetrieval
    - forward

## BlipForQuestionAnswering

[[autodoc]] BlipForQuestionAnswering
    - forward

</pt>
<tf>

## TFBlipModel

[[autodoc]] TFBlipModel
    - call
    - get_text_features
    - get_image_features

## TFBlipTextModel

[[autodoc]] TFBlipTextModel
    - call

## TFBlipVisionModel

[[autodoc]] TFBlipVisionModel
    - call

## TFBlipForConditionalGeneration

[[autodoc]] TFBlipForConditionalGeneration
    - call

## TFBlipForImageTextRetrieval

[[autodoc]] TFBlipForImageTextRetrieval
    - call

## TFBlipForQuestionAnswering

[[autodoc]] TFBlipForQuestionAnswering
    - call
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "a9fdaa06-a738-40a5-b907-e11e56ce676e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "8c0009bbca70921622cf3e8b87a72109af7024e115606242d3bb7c90a5f1604c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e7f4830-22e6-4a53-81d2-f948d9c1fab4", "node_type": "1", "metadata": {}, "hash": "f71057f60e0b274766089ee37f984314fd3ac146e8cb211542163f6dca0c3bbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2719, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Persimmon

## Overview

The Persimmon model was created by [ADEPT](https://www.adept.ai/blog/persimmon-8b), and authored by Erich Elsen, Augustus Odena, Maxwell Nye, Sağnak Taşırlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, Arushi Somani.

The authors introduced Persimmon-8B, a decoder model based on the classic transformers architecture, with query and key normalization. Persimmon-8B is a fully permissively-licensed model with approximately 8 billion parameters, released under the Apache license.  Some of the key attributes of Persimmon-8B are long context size (16K), performance, and capabilities for multimodal extensions.

The authors showcase their approach to model evaluation, focusing on practical text generation, mirroring how users interact with language models. The work also includes a comparative analysis, pitting Persimmon-8B against other prominent models (MPT 7B Instruct and Llama 2 Base 7B 1-Shot), across various evaluation tasks. The results demonstrate Persimmon-8B's competitive performance, even with limited training data.

In terms of model details, the work outlines the architecture and training methodology of Persimmon-8B, providing insights into its design choices, sequence length, and dataset composition. The authors present a fast inference code that outperforms traditional implementations through operator fusion and CUDA graph utilization while maintaining code coherence. They express their anticipation of how the community will leverage this contribution to drive innovation, hinting at further upcoming releases as part of an ongoing series of developments.

This model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).
The original code can be found [here](https://github.com/persimmon-ai-labs/adept-inference).

## Usage tips

<Tip warning={true}>

The `Persimmon` models were trained using `bfloat16`, but the original inference uses `float16` The checkpoints uploaded on the hub use `torch_dtype = 'float16'` which will be
used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. 

The `dtype` of the online weights is mostly irrelevant, unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online) then it will be cast to the default `dtype` of `torch` (becomes `torch.float32`). Users should specify the `torch_dtype` they want, and if they don't it will be `torch.float32`.

Finetuning the model in `float16` is not recommended and known to produce `nan`, as such the model should be fine-tuned in `bfloat16`.

</Tip>


key: _node_content
string_value:
{"id_": "9e7f4830-22e6-4a53-81d2-f948d9c1fab4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "8c0009bbca70921622cf3e8b87a72109af7024e115606242d3bb7c90a5f1604c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9fdaa06-a738-40a5-b907-e11e56ce676e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers"}, "hash": "e3c89142ff776d0a8a7ba39fa83d12c343034d00e039a55394235ac01c89dc22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2722, "end_char_idx": 4708, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Tips:

- To convert the model, you need to clone the original repository using `git clone https://github.com/persimmon-ai-labs/adept-inference`, then get the checkpoints:

```bash
git clone https://github.com/persimmon-ai-labs/adept-inference
wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_base_model_release.tar
tar -xvf 8b_base_model_release.tar
python src/transformers/models/persimmon/convert_persimmon_weights_to_hf.py  --input_dir /path/to/downloaded/persimmon/weights/ --output_dir /output/path \
    --pt_model_path /path/to/8b_chat_model_release/iter_0001251/mp_rank_00/model_optim_rng.pt
    --ada_lib_path /path/to/adept-inference
```

For the chat model:
```bash
wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar
tar -xvf 8b_base_model_release.tar
```

Thereafter, models can be loaded via:

```py
from transformers import PersimmonForCausalLM, PersimmonTokenizer

model = PersimmonForCausalLM.from_pretrained("/output/path")
tokenizer = PersimmonTokenizer.from_pretrained("/output/path")
```


- Perismmon uses a `sentencepiece` based tokenizer, with a `Unigram` model. It supports bytefallback, which is only available in `tokenizers==0.14.0` for the fast tokenizer.
The `LlamaTokenizer` is used as it is a standard wrapper around sentencepiece. The `chat` template will be updated with the templating functions in a follow up PR!

- The authors suggest to use the following prompt format for the chat mode: `f"human: {prompt}\n\nadept:"`


## PersimmonConfig

[[autodoc]] PersimmonConfig

## PersimmonModel

[[autodoc]] PersimmonModel
    - forward

## PersimmonForCausalLM

[[autodoc]] PersimmonForCausalLM
    - forward

## PersimmonForSequenceClassification

[[autodoc]] PersimmonForSequenceClassification
    - forward

## PersimmonForTokenClassification

[[autodoc]] PersimmonForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "0a31ad89-8443-4754-8c17-9034d64df1ed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "7c340fd48d6da01aea66faa86738414e5e95cd3647e94d8820eb0862a3175b07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c602e5c-790e-4401-88ed-1d2a557020ea", "node_type": "1", "metadata": {}, "hash": "bf66f3b03383cac5b0b17b826cf800ed42b73ac9bb35089422818eb6dc5dd505", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DePlot

## Overview 

DePlot was proposed in the paper [DePlot: One-shot visual language reasoning by plot-to-table translation](https://arxiv.org/abs/2212.10505) from Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun.

The abstract of the paper states the following:

*Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24.0% improvement over finetuned SOTA on human-written queries from the task of chart QA.*

DePlot is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).
DePlot is a Visual Question Answering subset of `Pix2Struct` architecture. It renders the input question on the image and predicts the answer.

## Usage example

Currently one checkpoint is available for DePlot:

- `google/deplot`: DePlot fine-tuned on ChartQA dataset


key: _node_content
string_value:
{"id_": "2c602e5c-790e-4401-88ed-1d2a557020ea", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "7c340fd48d6da01aea66faa86738414e5e95cd3647e94d8820eb0862a3175b07", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a31ad89-8443-4754-8c17-9034d64df1ed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers"}, "hash": "821d0ab478cf69752ce1806e5654d3883f5465bfc4d6b9b74efa6efae4f9e2f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2171, "end_char_idx": 3610, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import AutoProcessor, Pix2StructForConditionalGeneration
import requests
from PIL import Image

model = Pix2StructForConditionalGeneration.from_pretrained("google/deplot")
processor = AutoProcessor.from_pretrained("google/deplot")
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))
```

## Fine-tuning

To fine-tune DePlot, refer to the pix2struct [fine-tuning notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb). For `Pix2Struct` models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faster convergence:
```python
from transformers.optimization import Adafactor, get_cosine_schedule_with_warmup

optimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)
scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)
```

<Tip>

DePlot is a model trained using `Pix2Struct` architecture. For API reference, see [`Pix2Struct` documentation](pix2struct).

</Tip>


key: _node_content
string_value:
{"id_": "ebcb403f-253c-4113-aec6-5b09c92eed7e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gemma2", "title": "Gemma2", "tokens": 642, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "371b66bb-8bf8-4c1d-81a0-a2a20cc13b28", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/gemma2", "title": "Gemma2", "tokens": 642, "retrieve_doc": true, "source": "transformers"}, "hash": "c57ad0e67593dd7b3ee85dbf606c8d01a19dda91ed0b9654f95b4db3949a6934", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1918, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Gemma2

## Overview

The Gemma2 model was proposed in [Gemma2: Open Models Based on Gemini Technology and Research](https://blog.google/technology/developers/google-gemma-2/) by Gemma2 Team, Google.
Two Gemma2 models are released, with parameters sizes of 9 billion (9B) and 27 billion (27B).

The abstract from the blog post is the following:

*Now we’re officially releasing Gemma 2 to researchers and developers globally. Available in both 9 billion (9B) and 27 billion (27B) parameter sizes, Gemma 2 is higher-performing and more efficient at inference than the first generation, with significant safety advancements built in. In fact, at 27B, it offers competitive alternatives to models more than twice its size, delivering the kind of performance that was only possible with proprietary models as recently as December.*

Tips:

- The original checkpoints can be converted using the conversion script `src/transformers/models/Gemma2/convert_Gemma2_weights_to_hf.py` 

<Tip warning={true}>

- Gemma2 uses sliding window attention every second layer, which makes it unsuitable for typical kv caching with [`~DynamicCache`] or tuples of tensors. To enable caching in Gemma2 forward call, you must initialize a [`~HybridCache`] instance and pass it as `past_key_values` to the forward call. Note, that you also have to prepare `cache_position` if the `past_key_values` already contains previous keys and values.

</Tip>

This model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Pedro Cuenca](https://huggingface.co/pcuenq) and [Tom Arsen]().


## Gemma2Config

[[autodoc]] Gemma2Config

## Gemma2Model

[[autodoc]] Gemma2Model
    - forward

## Gemma2ForCausalLM

[[autodoc]] Gemma2ForCausalLM
    - forward

## Gemma2ForSequenceClassification

[[autodoc]] Gemma2ForSequenceClassification
    - forward

## Gemma2ForTokenClassification

[[autodoc]] Gemma2ForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "4573eb41-2177-430f-8e96-08d7806d5cb4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers"}, "hash": "959aef429e82a136fd83c426f0985cc946b82991b716e40bbf66ed1017be7c3e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c97be010-b686-4efc-b731-616ac1c34823", "node_type": "1", "metadata": {}, "hash": "21600adbb10d029becae8e0ef62881c7d29a3ba2dfaba20c69ce1df8e54ecd2a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3519, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MaskFormer

<Tip>

This is a recently introduced model so the API hasn't been tested extensively. There may be some bugs or slight
breaking changes to fix it in the future. If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title).

</Tip>

## Overview

The MaskFormer model was proposed in [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://arxiv.org/abs/2107.06278) by Bowen Cheng, Alexander G. Schwing, Alexander Kirillov. MaskFormer addresses semantic segmentation with a mask classification paradigm instead of performing classic pixel-level classification.

The abstract from the paper is the following:

*Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.*

The figure below illustrates the architecture of MaskFormer. Taken from the [original paper](https://arxiv.org/abs/2107.06278).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/maskformer_architecture.png"/>

This model was contributed by [francesco](https://huggingface.co/francesco). The original code can be found [here](https://github.com/facebookresearch/MaskFormer).

## Usage tips

-  MaskFormer's Transformer decoder is identical to the decoder of [DETR](detr). During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter `use_auxiliary_loss` of [`MaskFormerConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters).
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  `get_num_masks` function inside in the `MaskFormerLoss` class of `modeling_maskformer.py`. When training on multiple nodes, this should be
  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169).
- One can use [`MaskFormerImageProcessor`] to prepare images for the model and optional targets for the model.
- To get the final segmentation, depending on the task, you can call [`~MaskFormerImageProcessor.post_process_semantic_segmentation`] or [`~MaskFormerImageProcessor.post_process_panoptic_segmentation`].


key: _node_content
string_value:
{"id_": "c97be010-b686-4efc-b731-616ac1c34823", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers"}, "hash": "959aef429e82a136fd83c426f0985cc946b82991b716e40bbf66ed1017be7c3e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4573eb41-2177-430f-8e96-08d7806d5cb4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers"}, "hash": "5bedfbe5c3d922ecd98e32840fc18e83112fc411491cb5f2d2ff0de18d3de600", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1848, "end_char_idx": 5083, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Taken from the [original paper](https://arxiv.org/abs/2107.06278).

<img width="600" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/maskformer_architecture.png"/>

This model was contributed by [francesco](https://huggingface.co/francesco). The original code can be found [here](https://github.com/facebookresearch/MaskFormer).

## Usage tips

-  MaskFormer's Transformer decoder is identical to the decoder of [DETR](detr). During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter `use_auxiliary_loss` of [`MaskFormerConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters).
- If you want to train the model in a distributed environment across multiple nodes, then one should update the
  `get_num_masks` function inside in the `MaskFormerLoss` class of `modeling_maskformer.py`. When training on multiple nodes, this should be
  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169).
- One can use [`MaskFormerImageProcessor`] to prepare images for the model and optional targets for the model.
- To get the final segmentation, depending on the task, you can call [`~MaskFormerImageProcessor.post_process_semantic_segmentation`] or [`~MaskFormerImageProcessor.post_process_panoptic_segmentation`]. Both tasks can be solved using [`MaskFormerForInstanceSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.

## Resources

<PipelineTag pipeline="image-segmentation"/>

- All notebooks that illustrate inference as well as fine-tuning on custom data with MaskFormer can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MaskFormer).
- Scripts for finetuning [`MaskFormer`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).

## MaskFormer specific outputs

[[autodoc]] models.maskformer.modeling_maskformer.MaskFormerModelOutput

[[autodoc]] models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput

## MaskFormerConfig

[[autodoc]] MaskFormerConfig

## MaskFormerImageProcessor

[[autodoc]] MaskFormerImageProcessor
    - preprocess
    - encode_inputs
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## MaskFormerFeatureExtractor

[[autodoc]] MaskFormerFeatureExtractor
    - __call__
    - encode_inputs
    - post_process_semantic_segmentation
    - post_process_instance_segmentation
    - post_process_panoptic_segmentation

## MaskFormerModel

[[autodoc]] MaskFormerModel
    - forward

## MaskFormerForInstanceSegmentation

[[autodoc]] MaskFormerForInstanceSegmentation
    - forward


