# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 05404693-f199-4e83-a337-c0092096dbe5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"05404693-f199-4e83-a337-c0092096dbe5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1561756b-cc39-435e-9962-9cf71dd1ec38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5526658e-858c-4b78-8c51-9cd9f2787469\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2cb92c43baff8cdd69ec61a16e32c25a6d1b7cbb4227a60e4f8e814b1895688d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"062af563-5753-4101-a5a0-ef087b71a5ef\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ec107b5b570974fe4a806e8de33ce2b70d0cea59aa5d31c6e08ca3cfcba2ef4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3839, \"end_char_idx\": 7658, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "ref_doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "chroma:document": "##### Usage\n\nTo load a model using Flash Attention 2, we can pass the `attn_implementation=\"flash_attention_2\"` flag to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n```\n\n##### Performance comparison\n\n\nThe following diagram shows the latency for the native attention implementation (no optimisation) against Better Transformer and Flash Attention 2. In all cases, we generate 400 semantic tokens on a 40GB A100 GPU with PyTorch 2.1. Flash Attention 2 is also consistently faster than Better Transformer, and its performance improves even more as batch sizes increase:\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ylacombe/benchmark-comparison/resolve/main/Bark%20Optimization%20Benchmark.png\">\n</div>\n\nTo put this into perspective, on an NVIDIA A100 and when generating 400 semantic tokens with a batch size of 16, you can get 17 times the [throughput](https://huggingface.co/blog/optimizing-bark#throughput) and still be 2 seconds faster than generating sentences one by one with the native model implementation. In other words, all the samples will be generated 17 times faster.\n\nAt batch size 8, on an NVIDIA A100, Flash Attention 2 is also 10% faster than Better Transformer, and at batch size 16, 25%.\n\n\n#### Combining optimization techniques\n\nYou can combine optimization techniques, and use CPU offload, half-precision and Flash Attention 2 (or \ud83e\udd17 Better Transformer) all at once.\n\n```python\nfrom transformers import BarkModel\nimport torch\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n# load in fp16 and use Flash Attention 2\nmodel = BarkModel.from_pretrained(\"suno/bark-small\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n\n# enable CPU offload\nmodel.enable_cpu_offload()\n```\n\nFind out more on inference optimization techniques [here](https://huggingface.co/docs/transformers/perf_infer_gpu_one).\n\n### Usage tips\n\nSuno offers a library of voice presets in a number of languages [here](https://suno-ai.notion.site/8b8e8749ed514b0cbf3f699013548683?v=bc67cff786b04b50b3ceb756fd05f68c).\nThese presets are also uploaded in the hub [here](https://huggingface.co/suno/bark-small/tree/main/speaker_embeddings) or [here](https://huggingface.co/suno/bark/tree/main/speaker_embeddings).\n\n```python\n>>> from transformers import AutoProcessor, BarkModel\n\n>>> processor = AutoProcessor.from_pretrained(\"suno/bark\")\n>>> model = BarkModel.from_pretrained(\"suno/bark\")\n\n>>> voice_preset = \"v2/en_speaker_6\"\n\n>>> inputs = processor(\"Hello, my dog is cute\", voice_preset=voice_preset)\n\n>>> audio_array = model.generate(**inputs)\n>>> audio_array = audio_array.cpu().numpy().squeeze()\n```\n\nBark can generate highly realistic, **multilingual** speech as well as other audio - including music, background noise and simple sound effects. \n\n```python\n>>> # Multilingual speech - simplified Chinese\n>>> inputs = processor(\"\u60ca\u4eba\u7684\uff01\u6211\u4f1a\u8bf4\u4e2d\u6587\")\n\n>>> # Multilingual speech - French - let's use a voice_preset as well\n>>> inputs = processor(\"Incroyable! Je peux g\u00e9n\u00e9rer du son.\", voice_preset=\"fr_speaker_5\")\n\n>>> # Bark can also generate music. You can help it out by adding music notes around your lyrics.\n>>> inputs = processor(\"\u266a Hello, my dog is cute \u266a\")\n\n>>> audio_array = model.generate(**inputs)\n>>> audio_array = audio_array.cpu().numpy().squeeze()\n```\n\nThe model can also produce **nonverbal communications** like laughing, sighing and crying."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 062af563-5753-4101-a5a0-ef087b71a5ef
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bark", "title": "Bark", "tokens": 2279, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"062af563-5753-4101-a5a0-ef087b71a5ef\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1561756b-cc39-435e-9962-9cf71dd1ec38\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e023131db50dd78a4993c78bf26a853014d36071eb90afe90b5db0b4ad24cf1d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05404693-f199-4e83-a337-c0092096dbe5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bark\", \"title\": \"Bark\", \"tokens\": 2279, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"817762c7b856ebfc08f2649840eedecbe664779483f7169b62e348e54dd74a3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7661, \"end_char_idx\": 8890, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "ref_doc_id": "1561756b-cc39-435e-9962-9cf71dd1ec38", "chroma:document": "```python\n>>> # Adding non-speech cues to the input text\n>>> inputs = processor(\"Hello uh ... [clears throat], my dog is cute [laughter]\")\n\n>>> audio_array = model.generate(**inputs)\n>>> audio_array = audio_array.cpu().numpy().squeeze()\n```\n\nTo save the audio, simply take the sample rate from the model config and some scipy utility:\n\n```python\n>>> from scipy.io.wavfile import write as write_wav\n\n>>> # save audio to disk, but first take the sample rate from the model config\n>>> sample_rate = model.generation_config.sample_rate\n>>> write_wav(\"bark_generation.wav\", sample_rate, audio_array)\n```\n\n## BarkConfig\n\n[[autodoc]] BarkConfig\n    - all\n\n## BarkProcessor\n\n[[autodoc]] BarkProcessor\n    - all\n    - __call__\n\n## BarkModel\n\n[[autodoc]] BarkModel\n    - generate\n    - enable_cpu_offload\n\n## BarkSemanticModel\n\n[[autodoc]] BarkSemanticModel\n    - forward\n\n## BarkCoarseModel\n\n[[autodoc]] BarkCoarseModel\n    - forward\n\n## BarkFineModel\n\n[[autodoc]] BarkFineModel\n    - forward\n\n## BarkCausalModel\n\n[[autodoc]] BarkCausalModel\n    - forward\n\n## BarkCoarseConfig\n\n[[autodoc]] BarkCoarseConfig\n    - all\n\n## BarkFineConfig\n\n[[autodoc]] BarkFineConfig\n    - all\n\n## BarkSemanticConfig\n\n[[autodoc]] BarkSemanticConfig\n    - all"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a601726e-3fb7-437a-bc5a-f344f18e42c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a601726e-3fb7-437a-bc5a-f344f18e42c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81dc3773-a83b-4b36-ab50-bbd3ad06fd1a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c132dc5980022b1c3c3eb0cccd19de9d7dd59841cb2e25ce19d9b0a3e0bce6e4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"32eade85-ccda-4557-aaf1-aa68c15b6550\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"623b37112bb4f8c2164042414adf8ad56ea8be22598a1cabb170134a86e6ac2c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3267, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "doc_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "ref_doc_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "chroma:document": "# ErnieM\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe ErnieM model was proposed in [ERNIE-M: Enhanced Multilingual Representation by Aligning\nCross-lingual Semantics with Monolingual Corpora](https://arxiv.org/abs/2012.15674)  by Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun,\nHao Tian, Hua Wu, Haifeng Wang.\n\nThe abstract from the paper is the following:\n\n*Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for lowresource languages. In this paper, we propose ERNIE-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks.*\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato). The original code can be found [here](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers/ernie_m).\n\n\n## Usage tips\n\n- Ernie-M is a BERT-like model so it is a stacked Transformer Encoder.\n- Instead of using MaskedLM for pretraining (like BERT) the authors used two novel techniques: `Cross-attention Masked Language Modeling` and `Back-translation Masked Language Modeling`. For now these two LMHead objectives are not implemented here.\n- It is a multilingual language model.\n- Next Sentence Prediction was not used in pretraining process.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ErnieMConfig\n\n[[autodoc]] ErnieMConfig\n\n\n## ErnieMTokenizer\n\n[[autodoc]] ErnieMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n\n## ErnieMModel\n\n[[autodoc]] ErnieMModel\n    - forward\n\n## ErnieMForSequenceClassification\n\n[[autodoc]] ErnieMForSequenceClassification\n    - forward\n\n\n## ErnieMForMultipleChoice\n\n[[autodoc]] ErnieMForMultipleChoice\n    - forward\n\n\n## ErnieMForTokenClassification\n\n[[autodoc]] ErnieMForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 32eade85-ccda-4557-aaf1-aa68c15b6550
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/ernie_m", "title": "ErnieM", "tokens": 970, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"32eade85-ccda-4557-aaf1-aa68c15b6550\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"81dc3773-a83b-4b36-ab50-bbd3ad06fd1a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c132dc5980022b1c3c3eb0cccd19de9d7dd59841cb2e25ce19d9b0a3e0bce6e4\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a601726e-3fb7-437a-bc5a-f344f18e42c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/ernie_m\", \"title\": \"ErnieM\", \"tokens\": 970, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4993d7f8dc72344bbba71ed3a904493d30cd2e7c1af5d98bd8997e45f73c3814\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2011, \"end_char_idx\": 3446, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "doc_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "ref_doc_id": "81dc3773-a83b-4b36-ab50-bbd3ad06fd1a", "chroma:document": "## Usage tips\n\n- Ernie-M is a BERT-like model so it is a stacked Transformer Encoder.\n- Instead of using MaskedLM for pretraining (like BERT) the authors used two novel techniques: `Cross-attention Masked Language Modeling` and `Back-translation Masked Language Modeling`. For now these two LMHead objectives are not implemented here.\n- It is a multilingual language model.\n- Next Sentence Prediction was not used in pretraining process.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ErnieMConfig\n\n[[autodoc]] ErnieMConfig\n\n\n## ErnieMTokenizer\n\n[[autodoc]] ErnieMTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n\n## ErnieMModel\n\n[[autodoc]] ErnieMModel\n    - forward\n\n## ErnieMForSequenceClassification\n\n[[autodoc]] ErnieMForSequenceClassification\n    - forward\n\n\n## ErnieMForMultipleChoice\n\n[[autodoc]] ErnieMForMultipleChoice\n    - forward\n\n\n## ErnieMForTokenClassification\n\n[[autodoc]] ErnieMForTokenClassification\n    - forward\n\n\n## ErnieMForQuestionAnswering\n\n[[autodoc]] ErnieMForQuestionAnswering\n    - forward\n\n## ErnieMForInformationExtraction\n\n[[autodoc]] ErnieMForInformationExtraction\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 18d78e40-87b9-402d-a50b-755fc8163010
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"18d78e40-87b9-402d-a50b-755fc8163010\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75ba7b0c-8030-4991-8409-30f1f9cde43c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9710c0628b0c252fe9fb1c339e373efe622fa35a4dc4f0ef0decb79e1c84ea4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"84b5362e-1d63-4908-b892-9c14bfd0bc29\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"050bdc5091aae5f3ec9e79e94af8e9a1e3d00366402b636df996b49b515483db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3280, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "doc_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "ref_doc_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "chroma:document": "# SegGPT\n\n## Overview\n\nThe SegGPT model was proposed in [SegGPT: Segmenting Everything In Context](https://arxiv.org/abs/2304.03284) by Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, Tiejun Huang. SegGPT employs a decoder-only Transformer that can generate a segmentation mask given an input image, a prompt image and its corresponding prompt mask. The model achieves remarkable one-shot results with 56.1 mIoU on COCO-20 and 85.6 mIoU on FSS-1000.\n\nThe abstract from the paper is the following:\n\n*We present SegGPT, a generalist model for segmenting everything in context. We unify various segmentation tasks into a generalist in-context learning framework that accommodates different kinds of segmentation data by transforming them into the same format of images. The training of SegGPT is formulated as an in-context coloring problem with random color mapping for each data sample. The objective is to accomplish diverse tasks according to the context, rather than relying on specific colors. After training, SegGPT can perform arbitrary segmentation tasks in images or videos via in-context inference, such as object instance, stuff, part, contour, and text. SegGPT is evaluated on a broad range of tasks, including few-shot semantic segmentation, video object segmentation, semantic segmentation, and panoptic segmentation. Our results show strong capabilities in segmenting in-domain and out-of*\n\nTips:\n- One can use [`SegGptImageProcessor`] to prepare image input, prompt and mask to the model.\n- One can either use segmentation maps or RGB images as prompt masks. If using the latter make sure to set `do_convert_rgb=False` in the `preprocess` method.\n- It's highly advisable to pass `num_labels` when using `segmetantion_maps` (not considering background) during preprocessing and postprocessing with [`SegGptImageProcessor`] for your use case.\n- When doing inference with [`SegGptForImageSegmentation`] if your `batch_size` is greater than 1 you can use feature ensemble across your images by passing `feature_ensemble=True` in the forward method.\n\nHere's how to use the model for one-shot semantic segmentation:\n\n```python\nimport torch\nfrom datasets import load_dataset\nfrom transformers import SegGptImageProcessor, SegGptForImageSegmentation\n\ncheckpoint = \"BAAI/seggpt-vit-large\"\nimage_processor = SegGptImageProcessor.from_pretrained(checkpoint)\nmodel = SegGptForImageSegmentation.from_pretrained(checkpoint)\n\ndataset_id = \"EduardoPacheco/FoodSeg103\"\nds = load_dataset(dataset_id, split=\"train\")\n# Number of labels in FoodSeg103 (not including background)\nnum_labels = 103\n\nimage_input = ds[4][\"image\"]\nground_truth = ds[4][\"label\"]\nimage_prompt = ds[29][\"image\"]\nmask_prompt = ds[29][\"label\"]\n\ninputs = image_processor(\n    images=image_input, \n    prompt_images=image_prompt,\n    segmentation_maps=mask_prompt, \n    num_labels=num_labels,\n    return_tensors=\"pt\"\n)\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\ntarget_sizes = [image_input.size[::-1]]\nmask = image_processor.post_process_semantic_segmentation(outputs, target_sizes, num_labels=num_labels)[0]\n```\n\nThis model was contributed by [EduardoPacheco](https://huggingface.co/EduardoPacheco).\nThe original code can be found [here]([(https://github.com/baaivision/Painter/tree/main))."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 84b5362e-1d63-4908-b892-9c14bfd0bc29
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seggpt", "title": "SegGPT", "tokens": 1012, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"84b5362e-1d63-4908-b892-9c14bfd0bc29\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75ba7b0c-8030-4991-8409-30f1f9cde43c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9710c0628b0c252fe9fb1c339e373efe622fa35a4dc4f0ef0decb79e1c84ea4a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18d78e40-87b9-402d-a50b-755fc8163010\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seggpt\", \"title\": \"SegGPT\", \"tokens\": 1012, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88329ac691b57817d5a4fe064260a73ce0f3e8b487c5e6907c4f2f2db099b812\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3283, \"end_char_idx\": 3581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "doc_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "ref_doc_id": "75ba7b0c-8030-4991-8409-30f1f9cde43c", "chroma:document": "## SegGptConfig\n\n[[autodoc]] SegGptConfig\n\n## SegGptImageProcessor\n\n[[autodoc]] SegGptImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n## SegGptModel\n\n[[autodoc]] SegGptModel\n    - forward\n\n## SegGptForImageSegmentation\n\n[[autodoc]] SegGptForImageSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a2174e29-3665-499a-aaef-bcb1632f29d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a2174e29-3665-499a-aaef-bcb1632f29d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1ba06b4f-e9fe-486e-801c-de4f88306799\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"412a0a25-9857-4292-a4ff-e15cbb8750c1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1102cfabdfdb4a0bb54c0fe097f018c19a4477b104572e2c3f9197f8e9765d7e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2641, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "doc_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "ref_doc_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "chroma:document": "# FastSpeech2Conformer\n\n## Overview\n\nThe FastSpeech2Conformer model was proposed with the paper [Recent Developments On Espnet Toolkit Boosted By Conformer](https://arxiv.org/abs/2010.13956) by Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang.\n\nThe abstract from the original FastSpeech2 paper is the following:\n\n*Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.*\n\nThis model was contributed by [Connor Henderson](https://huggingface.co/connor-henderson). The original code can be found [here](https://github.com/espnet/espnet/blob/master/espnet2/tts/fastspeech2/fastspeech2.py)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 412a0a25-9857-4292-a4ff-e15cbb8750c1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"412a0a25-9857-4292-a4ff-e15cbb8750c1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1ba06b4f-e9fe-486e-801c-de4f88306799\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2174e29-3665-499a-aaef-bcb1632f29d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e13746a38df065a33fb502fdbe82b50b75495a593e1121047d344275b85e5680\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"13469cb5-9b57-4354-8f9e-1f0e5b818d04\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b54c25d46b697cf2fef25d9b174c94ae3029eff0ee79f865f65a64ece432c581\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2644, \"end_char_idx\": 5598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "doc_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "ref_doc_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "chroma:document": "## \ud83e\udd17 Model Architecture\nFastSpeech2's general structure with a Mel-spectrogram decoder was implemented, and the traditional transformer blocks were replaced with conformer blocks as done in the ESPnet library.\n\n#### FastSpeech2 Model Architecture\n![FastSpeech2 Model Architecture](https://www.microsoft.com/en-us/research/uploads/prod/2021/04/fastspeech2-1.png)\n\n#### Conformer Blocks\n![Conformer Blocks](https://www.researchgate.net/profile/Hirofumi-Inaguma-2/publication/344911155/figure/fig2/AS:951455406108673@1603856054097/An-overview-of-Conformer-block.png)\n\n#### Convolution Module\n![Convolution Module](https://d3i71xaburhd42.cloudfront.net/8809d0732f6147d4ad9218c8f9b20227c837a746/2-Figure1-1.png)\n\n## \ud83e\udd17 Transformers Usage\n\nYou can run FastSpeech2Conformer locally with the \ud83e\udd17 Transformers library.\n\n1. First install the \ud83e\udd17 [Transformers library](https://github.com/huggingface/transformers), g2p-en:\n\n```bash\npip install --upgrade pip\npip install --upgrade transformers g2p-en\n```\n\n2. Run inference via the Transformers modelling code with the model and hifigan separately\n\n```python\n\nfrom transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerModel, FastSpeech2ConformerHifiGan\nimport soundfile as sf\n\ntokenizer = FastSpeech2ConformerTokenizer.from_pretrained(\"espnet/fastspeech2_conformer\")\ninputs = tokenizer(\"Hello, my dog is cute.\", return_tensors=\"pt\")\ninput_ids = inputs[\"input_ids\"]\n\nmodel = FastSpeech2ConformerModel.from_pretrained(\"espnet/fastspeech2_conformer\")\noutput_dict = model(input_ids, return_dict=True)\nspectrogram = output_dict[\"spectrogram\"]\n\nhifigan = FastSpeech2ConformerHifiGan.from_pretrained(\"espnet/fastspeech2_conformer_hifigan\")\nwaveform = hifigan(spectrogram)\n\nsf.write(\"speech.wav\", waveform.squeeze().detach().numpy(), samplerate=22050)\n```\n\n3. Run inference via the Transformers modelling code with the model and hifigan combined\n\n```python\nfrom transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan\nimport soundfile as sf\n\ntokenizer = FastSpeech2ConformerTokenizer.from_pretrained(\"espnet/fastspeech2_conformer\")\ninputs = tokenizer(\"Hello, my dog is cute.\", return_tensors=\"pt\")\ninput_ids = inputs[\"input_ids\"]\n\nmodel = FastSpeech2ConformerWithHifiGan.from_pretrained(\"espnet/fastspeech2_conformer_with_hifigan\")\noutput_dict = model(input_ids, return_dict=True)\nwaveform = output_dict[\"waveform\"]\n\nsf.write(\"speech.wav\", waveform.squeeze().detach().numpy(), samplerate=22050)\n```\n\n4. Run inference with a pipeline and specify which vocoder to use\n```python\nfrom transformers import pipeline, FastSpeech2ConformerHifiGan\nimport soundfile as sf\n\nvocoder = FastSpeech2ConformerHifiGan.from_pretrained(\"espnet/fastspeech2_conformer_hifigan\")\nsynthesiser = pipeline(model=\"espnet/fastspeech2_conformer\", vocoder=vocoder)\n\nspeech = synthesiser(\"Hello, my dog is cooler than you!\")\n\nsf.write(\"speech.wav\", speech[\"audio\"].squeeze(), samplerate=speech[\"sampling_rate\"])\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 13469cb5-9b57-4354-8f9e-1f0e5b818d04
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer", "title": "FastSpeech2Conformer", "tokens": 1686, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"13469cb5-9b57-4354-8f9e-1f0e5b818d04\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1ba06b4f-e9fe-486e-801c-de4f88306799\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"def2b861944e0ca72b775d1986a210926bf59970ec9000f1fa994b750ad6d4aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"412a0a25-9857-4292-a4ff-e15cbb8750c1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fastspeech2_conformer\", \"title\": \"FastSpeech2Conformer\", \"tokens\": 1686, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0cac8dbac6f52a98e686d55399f44368669e071cba661a3dd9c10cbe5e66f344\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5601, \"end_char_idx\": 6259, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "doc_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "ref_doc_id": "1ba06b4f-e9fe-486e-801c-de4f88306799", "chroma:document": "## FastSpeech2ConformerConfig\n\n[[autodoc]] FastSpeech2ConformerConfig\n\n## FastSpeech2ConformerHifiGanConfig\n\n[[autodoc]] FastSpeech2ConformerHifiGanConfig\n\n## FastSpeech2ConformerWithHifiGanConfig\n\n[[autodoc]] FastSpeech2ConformerWithHifiGanConfig\n\n## FastSpeech2ConformerTokenizer\n\n[[autodoc]] FastSpeech2ConformerTokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n\n## FastSpeech2ConformerModel\n\n[[autodoc]] FastSpeech2ConformerModel\n    - forward\n\n## FastSpeech2ConformerHifiGan\n\n[[autodoc]] FastSpeech2ConformerHifiGan\n    - forward\n\n## FastSpeech2ConformerWithHifiGan\n\n[[autodoc]] FastSpeech2ConformerWithHifiGan\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a4e64056-3c94-466a-bdde-cd36297fbd46
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a4e64056-3c94-466a-bdde-cd36297fbd46\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c5a2c1f-e81c-475a-b67c-959c18eb9093\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40a004e7c40d19e666beb2724c740fe98f61a3bdcd676171eff04e4a0d65d538\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46f06ca8-18fe-41dd-a5a3-b0b4ce484ba9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9ca49fb8a63ba71ce65d371bfd3ec931e89e75c573411baa4ba7bc3d15088ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3076, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "doc_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "ref_doc_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "chroma:document": "# X-CLIP\n\n## Overview\n\nThe X-CLIP model was proposed in [Expanding Language-Image Pretrained Models for General Video Recognition](https://arxiv.org/abs/2208.02816) by Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling.\nX-CLIP is a minimal extension of [CLIP](clip) for video. The model consists of a text encoder, a cross-frame vision encoder, a multi-frame integration Transformer, and a video-specific prompt generator.\n\nThe abstract from the paper is the following:\n\n*Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable \"zero-shot\" generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.*\n\nTips:\n\n- Usage of X-CLIP is identical to [CLIP](clip).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> X-CLIP architecture. Taken from the <a href=\"https://arxiv.org/abs/2208.02816\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/VideoX/tree/master/X-CLIP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with X-CLIP.\n\n- Demo notebooks for X-CLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 46f06ca8-18fe-41dd-a5a3-b0b4ce484ba9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xclip", "title": "X-CLIP", "tokens": 1005, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"46f06ca8-18fe-41dd-a5a3-b0b4ce484ba9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5c5a2c1f-e81c-475a-b67c-959c18eb9093\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"40a004e7c40d19e666beb2724c740fe98f61a3bdcd676171eff04e4a0d65d538\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a4e64056-3c94-466a-bdde-cd36297fbd46\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xclip\", \"title\": \"X-CLIP\", \"tokens\": 1005, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fc17b184a9f7f5dccb3f49c342e4d3265c05df0faa5a5bad7142aa4822097b80\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1488, \"end_char_idx\": 3525, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "doc_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "ref_doc_id": "5c5a2c1f-e81c-475a-b67c-959c18eb9093", "chroma:document": "Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited.*\n\nTips:\n\n- Usage of X-CLIP is identical to [CLIP](clip).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> X-CLIP architecture. Taken from the <a href=\"https://arxiv.org/abs/2208.02816\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/microsoft/VideoX/tree/master/X-CLIP).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with X-CLIP.\n\n- Demo notebooks for X-CLIP can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## XCLIPProcessor\n\n[[autodoc]] XCLIPProcessor\n\n## XCLIPConfig\n\n[[autodoc]] XCLIPConfig\n    - from_text_vision_configs\n\n## XCLIPTextConfig\n\n[[autodoc]] XCLIPTextConfig\n\n## XCLIPVisionConfig\n\n[[autodoc]] XCLIPVisionConfig\n\n## XCLIPModel\n\n[[autodoc]] XCLIPModel\n    - forward\n    - get_text_features\n    - get_video_features\n\n## XCLIPTextModel\n\n[[autodoc]] XCLIPTextModel\n    - forward\n\n## XCLIPVisionModel\n\n[[autodoc]] XCLIPVisionModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1a08bd05-7ae3-461c-b26d-0d2e7eb0f9d3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1a08bd05-7ae3-461c-b26d-0d2e7eb0f9d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3a42e257-090c-487d-bb7a-dd2db3fc19c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c5c17d0-dbe2-4aae-849c-3a761304a4e9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74dbb02ca2c248bbe5608882fd2c92217fb1857e165ac137047494e4e94ade01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3001, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "doc_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "ref_doc_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "chroma:document": "# VideoMAE\n\n## Overview\n\nThe VideoMAE model was proposed in [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://arxiv.org/abs/2203.12602) by Zhan Tong, Yibing Song, Jue Wang, Limin Wang.\nVideoMAE extends masked auto encoders ([MAE](vit_mae)) to video, claiming state-of-the-art performance on several video classification benchmarks.\n\nThe abstract from the paper is the following:\n\n*Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking and reconstruction. These simple designs turn out to be effective for overcoming information leakage caused by the temporal correlation during video reconstruction. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. This is partially ascribed to the challenging task of video reconstruction to enforce high-level structure learning. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videomae_architecture.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> VideoMAE pre-training. Taken from the <a href=\"https://arxiv.org/abs/2203.12602\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/MCG-NJU/VideoMAE).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3c5c17d0-dbe2-4aae-849c-3a761304a4e9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3c5c17d0-dbe2-4aae-849c-3a761304a4e9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3a42e257-090c-487d-bb7a-dd2db3fc19c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1a08bd05-7ae3-461c-b26d-0d2e7eb0f9d3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c8c73508872c436c77f109629f644f2dee085f32fe4b5c9769548c78c65b936b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c619bd65-ef11-42a5-b8e9-68e3c7f75c65\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26daa989fc42d04257acb5afb63fce97a5306762c8d57c0b8926d6290f19a639\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1542, \"end_char_idx\": 4683, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "doc_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "ref_doc_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "chroma:document": "Domain shift between pre-training and target datasets are important issues in SSVP. Notably, our VideoMAE with the vanilla ViT backbone can achieve 83.9% on Kinects-400, 75.3% on Something-Something V2, 90.8% on UCF101, and 61.1% on HMDB51 without using any extra data.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videomae_architecture.jpeg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> VideoMAE pre-training. Taken from the <a href=\"https://arxiv.org/abs/2203.12602\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/MCG-NJU/VideoMAE).\n\n## Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import VideoMAEForVideoClassification\nmodel = VideoMAEForVideoClassification.from_pretrained(\"MCG-NJU/videomae-base-finetuned-kinetics\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MCG-NJU/videomae-base-finetuned-kinetics` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        37 |                                        10 |                      3.7  |\n|            2 |                                        24 |                                        18 |                      1.33 |\n|            4 |                                        43 |                                        32 |                      1.34 |\n|            8 |                                        84 |                                        60 |                      1.4  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with VideoMAE. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c619bd65-ef11-42a5-b8e9-68e3c7f75c65
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/videomae", "title": "VideoMAE", "tokens": 1478, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c619bd65-ef11-42a5-b8e9-68e3c7f75c65\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3a42e257-090c-487d-bb7a-dd2db3fc19c0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"79b40e8e575b51be54a8686e025cdc2ff0c421c9fd139169d212f24d71bb1ca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c5c17d0-dbe2-4aae-849c-3a761304a4e9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/videomae\", \"title\": \"VideoMAE\", \"tokens\": 1478, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0197d4fba3265d40096d73ab77e5df56b831a3af7a01451f8bf95be4761dfbac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3003, \"end_char_idx\": 5655, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "doc_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "ref_doc_id": "3a42e257-090c-487d-bb7a-dd2db3fc19c0", "chroma:document": "```\nfrom transformers import VideoMAEForVideoClassification\nmodel = VideoMAEForVideoClassification.from_pretrained(\"MCG-NJU/videomae-base-finetuned-kinetics\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MCG-NJU/videomae-base-finetuned-kinetics` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        37 |                                        10 |                      3.7  |\n|            2 |                                        24 |                                        18 |                      1.33 |\n|            4 |                                        43 |                                        32 |                      1.34 |\n|            8 |                                        84 |                                        60 |                      1.4  |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with VideoMAE. If\nyou're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll\nreview it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n**Video classification**\n- [A notebook](https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb) that shows how\nto fine-tune a VideoMAE model on a custom dataset.\n- [Video classification task guide](../tasks/video_classification)\n- [A \ud83e\udd17 Space](https://huggingface.co/spaces/sayakpaul/video-classification-ucf101-subset) showing how to perform inference with a video classification model.\n\n## VideoMAEConfig\n\n[[autodoc]] VideoMAEConfig\n\n## VideoMAEFeatureExtractor\n\n[[autodoc]] VideoMAEFeatureExtractor\n    - __call__\n\n## VideoMAEImageProcessor\n\n[[autodoc]] VideoMAEImageProcessor\n    - preprocess\n\n## VideoMAEModel\n\n[[autodoc]] VideoMAEModel\n    - forward\n\n## VideoMAEForPreTraining\n\n`VideoMAEForPreTraining` includes the decoder on top for self-supervised pre-training.\n\n[[autodoc]] transformers.VideoMAEForPreTraining\n    - forward\n\n## VideoMAEForVideoClassification\n\n[[autodoc]] transformers.VideoMAEForVideoClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e2ec2ff4-a3f5-419e-817a-5b3626fc78a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e2ec2ff4-a3f5-419e-817a-5b3626fc78a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f3764660-d3ca-4cf1-99e0-97ff562eed5e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b5eb4a2d-b672-4143-8640-3af2462eec36\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f208690bb0a8145af4cd177ceefb3138f224890000facd968d3c493ff40d5ea9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3328, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "ref_doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "chroma:document": "# Vision Transformer (ViT)\n\n## Overview\n\nThe Vision Transformer (ViT) model was proposed in [An Image is Worth 16x16 Words: Transformers for Image Recognition\nat Scale](https://arxiv.org/abs/2010.11929) by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk\nWeissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob\nUszkoreit, Neil Houlsby. It's the first paper that successfully trains a Transformer encoder on ImageNet, attaining\nvery good results compared to familiar convolutional architectures.\n\nThe abstract from the paper is the following:\n\n*While the Transformer architecture has become the de-facto standard for natural language processing tasks, its\napplications to computer vision remain limited. In vision, attention is either applied in conjunction with\nconvolutional networks, or used to replace certain components of convolutional networks while keeping their overall\nstructure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to\nsequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of\ndata and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.),\nVision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring\nsubstantially fewer computational resources to train.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViT architecture. Taken from the <a href=\"https://arxiv.org/abs/2010.11929\">original paper.</a> </small>\n\nFollowing the original Vision Transformer, some follow-up works have been made:\n\n- [DeiT](deit) (Data-efficient Image Transformers) by Facebook AI. DeiT models are distilled vision transformers.\n  The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [`ViTModel`] or\n  [`ViTForImageClassification`]. There are 4 variants available (in 3 different sizes): *facebook/deit-tiny-patch16-224*,\n  *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and *facebook/deit-base-patch16-384*. Note that one should\n  use [`DeiTImageProcessor`] in order to prepare images for the model.\n\n- [BEiT](beit) (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained\n  vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a VQ-VAE.\n\n- DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using\n  the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting\n  objects, without having ever been trained to do so. DINO checkpoints can be found on the [hub](https://huggingface.co/models?other=dino).\n\n- [MAE](vit_mae) (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion\n  (75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms\n  supervised pre-training after fine-tuning."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b5eb4a2d-b672-4143-8640-3af2462eec36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b5eb4a2d-b672-4143-8640-3af2462eec36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f3764660-d3ca-4cf1-99e0-97ff562eed5e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e2ec2ff4-a3f5-419e-817a-5b3626fc78a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"23a8450c56f3d3c4fbaa0dca78fccb707024934bb661b690692bc71780c076fc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dcc57756-49e6-482a-be7d-7d3406ec0728\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0f3c80007d7fa609bd8b70a9b5d01f567f2249d9ee639c32bfa5762877ae9c14\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1900, \"end_char_idx\": 5083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "ref_doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "chroma:document": "DeiT models are distilled vision transformers.\n  The authors of DeiT also released more efficiently trained ViT models, which you can directly plug into [`ViTModel`] or\n  [`ViTForImageClassification`]. There are 4 variants available (in 3 different sizes): *facebook/deit-tiny-patch16-224*,\n  *facebook/deit-small-patch16-224*, *facebook/deit-base-patch16-224* and *facebook/deit-base-patch16-384*. Note that one should\n  use [`DeiTImageProcessor`] in order to prepare images for the model.\n\n- [BEiT](beit) (BERT pre-training of Image Transformers) by Microsoft Research. BEiT models outperform supervised pre-trained\n  vision transformers using a self-supervised method inspired by BERT (masked image modeling) and based on a VQ-VAE.\n\n- DINO (a method for self-supervised training of Vision Transformers) by Facebook AI. Vision Transformers trained using\n  the DINO method show very interesting properties not seen with convolutional models. They are capable of segmenting\n  objects, without having ever been trained to do so. DINO checkpoints can be found on the [hub](https://huggingface.co/models?other=dino).\n\n- [MAE](vit_mae) (Masked Autoencoders) by Facebook AI. By pre-training Vision Transformers to reconstruct pixel values for a high portion\n  (75%) of masked patches (using an asymmetric encoder-decoder architecture), the authors show that this simple method outperforms\n  supervised pre-training after fine-tuning.\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code (written in JAX) can be\nfound [here](https://github.com/google-research/vision_transformer).\n\nNote that we converted the weights from Ross Wightman's [timm library](https://github.com/rwightman/pytorch-image-models),\nwho already converted the weights from JAX to PyTorch. Credits go to him!\n\n## Usage tips\n\n- To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\n  which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can be\n  used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of\n  vectors to a standard Transformer encoder.\n- As the Vision Transformer expects each image to be of the same size (resolution), one can use\n  [`ViTImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `google/vit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=vit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-21k](http://www.image-net.org/) (a collection of\n  14 million images and 21k classes) only, or (2) also fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- The Vision Transformer was pre-trained using a resolution of 224x224."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dcc57756-49e6-482a-be7d-7d3406ec0728
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dcc57756-49e6-482a-be7d-7d3406ec0728\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f3764660-d3ca-4cf1-99e0-97ff562eed5e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b5eb4a2d-b672-4143-8640-3af2462eec36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7965715fc65dfd16e15c002fd79ecf5931458c52426f6aeb6734d5ac832f20b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7e45117-f6dd-4a51-8cfb-622ea7f19658\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f000ddf0efdd7ec24f787f5f3e31fecfb6fd76251f7e04e948cc7652721115a8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3513, \"end_char_idx\": 6673, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "ref_doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "chroma:document": "Note that we converted the weights from Ross Wightman's [timm library](https://github.com/rwightman/pytorch-image-models),\nwho already converted the weights from JAX to PyTorch. Credits go to him!\n\n## Usage tips\n\n- To feed images to the Transformer encoder, each image is split into a sequence of fixed-size non-overlapping patches,\n  which are then linearly embedded. A [CLS] token is added to serve as representation of an entire image, which can be\n  used for classification. The authors also add absolute position embeddings, and feed the resulting sequence of\n  vectors to a standard Transformer encoder.\n- As the Vision Transformer expects each image to be of the same size (resolution), one can use\n  [`ViTImageProcessor`] to resize (or rescale) and normalize images for the model.\n- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of\n  each checkpoint. For example, `google/vit-base-patch16-224` refers to a base-sized architecture with patch\n  resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the [hub](https://huggingface.co/models?search=vit).\n- The available checkpoints are either (1) pre-trained on [ImageNet-21k](http://www.image-net.org/) (a collection of\n  14 million images and 21k classes) only, or (2) also fine-tuned on [ImageNet](http://www.image-net.org/challenges/LSVRC/2012/) (also referred to as ILSVRC 2012, a collection of 1.3 million\n  images and 1,000 classes).\n- The Vision Transformer was pre-trained using a resolution of 224x224. During fine-tuning, it is often beneficial to\n  use a higher resolution than pre-training [(Touvron et al., 2019)](https://arxiv.org/abs/1906.06423), [(Kolesnikov\n  et al., 2020)](https://arxiv.org/abs/1912.11370). In order to fine-tune at higher resolution, the authors perform\n  2D interpolation of the pre-trained position embeddings, according to their location in the original image.\n- The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed\n  an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked\n  language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant\n  improvement of 2% to training from scratch, but still 4% behind supervised pre-training.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d7e45117-f6dd-4a51-8cfb-622ea7f19658
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d7e45117-f6dd-4a51-8cfb-622ea7f19658\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f3764660-d3ca-4cf1-99e0-97ff562eed5e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dcc57756-49e6-482a-be7d-7d3406ec0728\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e20ca46aa6aa681bf39b5c915a2a9069f90ab05bb8c01b964cdf9b40826f3987\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ebf77dc0-e7e1-4f52-b8fd-de830060862d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c7d150dd84c9eb6b0fa496bbcd10457e8a7789d2ee87cbd5fb641d2e832843eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5084, \"end_char_idx\": 8496, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "ref_doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "chroma:document": "During fine-tuning, it is often beneficial to\n  use a higher resolution than pre-training [(Touvron et al., 2019)](https://arxiv.org/abs/1906.06423), [(Kolesnikov\n  et al., 2020)](https://arxiv.org/abs/1912.11370). In order to fine-tune at higher resolution, the authors perform\n  2D interpolation of the pre-trained position embeddings, according to their location in the original image.\n- The best results are obtained with supervised pre-training, which is not the case in NLP. The authors also performed\n  an experiment with a self-supervised pre-training objective, namely masked patched prediction (inspired by masked\n  language modeling). With this approach, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, a significant\n  improvement of 2% to training from scratch, but still 4% behind supervised pre-training.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ViTForImageClassification\nmodel = ViTForImageClassification.from_pretrained(\"google/vit-base-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-base-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         7 |                                         6 |                      1.17 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nDemo notebooks regarding inference as well as fine-tuning ViT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer).\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ebf77dc0-e7e1-4f52-b8fd-de830060862d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vit", "title": "Vision Transformer (ViT)", "tokens": 2874, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ebf77dc0-e7e1-4f52-b8fd-de830060862d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f3764660-d3ca-4cf1-99e0-97ff562eed5e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5279b9299cef45c6785f117e8d54b4a7583f5b9fcd9d1b7cf5dc7bc4d9831006\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7e45117-f6dd-4a51-8cfb-622ea7f19658\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vit\", \"title\": \"Vision Transformer (ViT)\", \"tokens\": 2874, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9b091568d32bc12de1944061c5fce4f5410ea18229b90ee83142fad3634591a9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6675, \"end_char_idx\": 11212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "ref_doc_id": "f3764660-d3ca-4cf1-99e0-97ff562eed5e", "chroma:document": "```\nfrom transformers import ViTForImageClassification\nmodel = ViTForImageClassification.from_pretrained(\"google/vit-base-patch16-224\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `google/vit-base-patch16-224` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                         7 |                                         6 |                      1.17 |\n|            2 |                                         8 |                                         6 |                      1.33 |\n|            4 |                                         8 |                                         6 |                      1.33 |\n|            8 |                                         8 |                                         6 |                      1.33 |\n\n## Resources\n\nDemo notebooks regarding inference as well as fine-tuning ViT on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/VisionTransformer).\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n`ViTForImageClassification` is supported by:\n<PipelineTag pipeline=\"image-classification\"/>\n\n- A blog post on how to [Fine-Tune ViT for Image Classification with Hugging Face Transformers](https://huggingface.co/blog/fine-tune-vit)\n- A blog post on [Image Classification with Hugging Face Transformers and `Keras`](https://www.philschmid.de/image-classification-huggingface-transformers-keras)\n- A notebook on [Fine-tuning for Image Classification with Hugging Face Transformers](https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb)\n- A notebook on how to [Fine-tune the Vision Transformer on CIFAR-10 with the Hugging Face Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb)\n- A notebook on how to [Fine-tune the Vision Transformer on CIFAR-10 with PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb)\n\n\u2697\ufe0f Optimization\n\n- A blog post on how to [Accelerate Vision Transformer (ViT) with Quantization using Optimum](https://www.philschmid.de/optimizing-vision-transformer)\n\n\u26a1\ufe0f Inference\n\n- A notebook on [Quick demo: Vision Transformer (ViT) by Google Brain](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Quick_demo_of_HuggingFace_version_of_Vision_Transformer_inference.ipynb)\n\n\ud83d\ude80 Deploy\n\n- A blog post on [Deploying Tensorflow Vision Models in Hugging Face with TF Serving](https://huggingface.co/blog/tf-serving-vision)\n- A blog post on [Deploying Hugging Face ViT on Vertex AI](https://huggingface.co/blog/deploy-vertex-ai)\n- A blog post on [Deploying Hugging Face ViT on Kubernetes with TF Serving](https://huggingface.co/blog/deploy-tfserving-kubernetes)\n\n## ViTConfig\n\n[[autodoc]] ViTConfig\n\n## ViTFeatureExtractor\n\n[[autodoc]] ViTFeatureExtractor\n    - __call__\n\n## ViTImageProcessor\n\n[[autodoc]] ViTImageProcessor\n    - preprocess\n\n## ViTImageProcessorFast\n\n[[autodoc]] ViTImageProcessorFast\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## ViTModel\n\n[[autodoc]] ViTModel\n    - forward\n\n## ViTForMaskedImageModeling\n\n[[autodoc]] ViTForMaskedImageModeling\n    - forward\n\n## ViTForImageClassification\n\n[[autodoc]] ViTForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFViTModel\n\n[[autodoc]] TFViTModel\n    - call\n\n## TFViTForImageClassification\n\n[[autodoc]] TFViTForImageClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxVitModel\n\n[[autodoc]] FlaxViTModel\n    - __call__\n\n## FlaxViTForImageClassification\n\n[[autodoc]] FlaxViTForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6d1590db-f71e-4a9d-bc34-2fbe397fc085
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6d1590db-f71e-4a9d-bc34-2fbe397fc085\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"651886c5-0a15-46d3-b53d-abcfae8b1fc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b25813ee-df36-4019-bb10-6707a178f8cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5f6d9d44d2a0a2dc8f7dff3d4017a86ae4e29dfd3a5d2a5baba17cb5056c4053\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3224, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "doc_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "ref_doc_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "chroma:document": "# Jamba\n\n## Overview\n\nJamba is a state-of-the-art, hybrid SSM-Transformer LLM. It is the first production-scale Mamba implementation, which opens up interesting research and application opportunities. While this initial experimentation shows encouraging gains, we expect these to be further enhanced with future optimizations and explorations.\n\nFor full details of this model please read the [release blog post](https://www.ai21.com/blog/announcing-jamba).\n\n### Model Details\n\nJamba is a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and an overall of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.\n\nAs depicted in the diagram below, Jamba's architecture features a blocks-and-layers approach that allows Jamba to successfully integrate Transformer and Mamba architectures altogether. Each Jamba block contains either an attention or a Mamba layer, followed by a multi-layer perceptron (MLP), producing an overall ratio of one Transformer layer out of every eight total layers.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/jamba_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n## Usage\n\n### Presequities\n\nJamba requires you use `transformers` version 4.39.0 or higher:\n```bash\npip install transformers>=4.39.0\n```\n\nIn order to run optimized Mamba implementations, you first need to install `mamba-ssm` and `causal-conv1d`:\n```bash\npip install mamba-ssm causal-conv1d>=1.2.0\n```\nYou also have to have the model on a CUDA device.\n\nYou can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.\n\n### Run the model\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\")\ntokenizer = AutoTokenizer.from_pretrained(\"ai21labs/Jamba-v0.1\")\n\ninput_ids = tokenizer(\"In the recent Super Bowl LVIII,\", return_tensors='pt').to(model.device)[\"input_ids\"]\n\noutputs = model.generate(input_ids, max_new_tokens=216)\n\nprint(tokenizer.batch_decode(outputs))\n# [\"<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\\n\\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\\n\\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\\n\\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b25813ee-df36-4019-bb10-6707a178f8cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b25813ee-df36-4019-bb10-6707a178f8cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"651886c5-0a15-46d3-b53d-abcfae8b1fc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6d1590db-f71e-4a9d-bc34-2fbe397fc085\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"33156d6de15c70296dce5b178196027bfee83783965ab70228b683788d48ecde\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ae2d17a9-0b31-4f88-a79b-194582372e42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"adbba3a1979e614f15bbb5d39c2e92546abab509692eecddb39203c978a41ee5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1611, \"end_char_idx\": 4816, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "doc_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "ref_doc_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "chroma:document": "You can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.\n\n### Run the model\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\")\ntokenizer = AutoTokenizer.from_pretrained(\"ai21labs/Jamba-v0.1\")\n\ninput_ids = tokenizer(\"In the recent Super Bowl LVIII,\", return_tensors='pt').to(model.device)[\"input_ids\"]\n\noutputs = model.generate(input_ids, max_new_tokens=216)\n\nprint(tokenizer.batch_decode(outputs))\n# [\"<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\\n\\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\\n\\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\\n\\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\\n\"]\n```\n\n<details>\n<summary><strong>Loading the model in half precision</strong></summary>\n\nThe published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16)\n# you can also use torch_dtype=torch.float16\n```\n\nWhen using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\",\n                                             torch_dtype=torch.bfloat16,\n                                             attn_implementation=\"flash_attention_2\",\n                                             device_map=\"auto\")\n```\n\n</details>\n<details><summary><strong>Load the model in 8-bit</strong></summary>\n\n**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ae2d17a9-0b31-4f88-a79b-194582372e42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/jamba", "title": "Jamba", "tokens": 1474, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ae2d17a9-0b31-4f88-a79b-194582372e42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"651886c5-0a15-46d3-b53d-abcfae8b1fc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e72895cf58c8f553e9b16e19ef52164eaef78505e21890ab1cb01a5ce3b8900c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b25813ee-df36-4019-bb10-6707a178f8cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/jamba\", \"title\": \"Jamba\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"102366a28ad619aa9e26c486b8e83e3d779cdc33a285b9b82b6fe92c618df6e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3124, \"end_char_idx\": 5556, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "doc_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "ref_doc_id": "651886c5-0a15-46d3-b53d-abcfae8b1fc8", "chroma:document": "This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\\n\"]\n```\n\n<details>\n<summary><strong>Loading the model in half precision</strong></summary>\n\nThe published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:\n\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16)\n# you can also use torch_dtype=torch.float16\n```\n\nWhen using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):\n```python\nfrom transformers import AutoModelForCausalLM\nimport torch\nmodel = AutoModelForCausalLM.from_pretrained(\"ai21labs/Jamba-v0.1\",\n                                             torch_dtype=torch.bfloat16,\n                                             attn_implementation=\"flash_attention_2\",\n                                             device_map=\"auto\")\n```\n\n</details>\n<details><summary><strong>Load the model in 8-bit</strong></summary>\n\n**Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index). In order to not degrade model quality, we recommend to exclude the Mamba blocks from the quantization:\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nquantization_config = BitsAndBytesConfig(load_in_8bit=True, llm_int8_skip_modules=[\"mamba\"])\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"ai21labs/Jamba-v0.1\", torch_dtype=torch.bfloat16, attn_implementation=\"flash_attention_2\", quantization_config=quantization_config\n)\n```\n</details>\n\n## JambaConfig\n\n[[autodoc]] JambaConfig\n\n\n## JambaModel\n\n[[autodoc]] JambaModel\n    - forward\n\n\n## JambaForCausalLM\n\n[[autodoc]] JambaForCausalLM\n    - forward\n\n\n## JambaForSequenceClassification\n\n[[autodoc]] transformers.JambaForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e1b773a9-3cd3-4f69-b746-96b06b0ddf3a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e1b773a9-3cd3-4f69-b746-96b06b0ddf3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"710e77e0-7b43-42d7-8a22-6c8e0274eb82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be9a8950-892a-4daa-bc83-ae3d9de68e09\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ba5d0c0df263d545be96b6e21a428d756dcc06322811b8f2408ab72258c6b33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "ref_doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "chroma:document": "# LLaMA\n\n## Overview\n\nThe LLaMA model was proposed in [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) by Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. It is a collection of foundation language models ranging from 7B to 65B parameters.\n\nThe abstract from the paper is the following:\n\n*We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community. *\n\nThis model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama).\n\n## Usage tips\n\n- Weights for the LLaMA models can be obtained from by filling out [this form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform?usp=send_form)\n- After downloading the weights, they will need to be converted to the Hugging Face Transformers format using the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py). The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: be9a8950-892a-4daa-bc83-ae3d9de68e09
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"be9a8950-892a-4daa-bc83-ae3d9de68e09\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"710e77e0-7b43-42d7-8a22-6c8e0274eb82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e1b773a9-3cd3-4f69-b746-96b06b0ddf3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"af06f8fb49fc311bd236a7d6d21388fa38dfe9d6785b5995f626d16c53f68882\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5b533443d7a5392adef42f8c712c0e6b5e43149fd5ac37142c44f2f4b5d94a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1846, \"end_char_idx\": 4707, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "ref_doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "chroma:document": "The script can be called with the following (example) command:\n\n```bash\npython src/transformers/models/llama/convert_llama_weights_to_hf.py \\\n    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path\n```\n\n- After conversion, the model and tokenizer can be loaded via:\n\n```python\nfrom transformers import LlamaForCausalLM, LlamaTokenizer\n\ntokenizer = LlamaTokenizer.from_pretrained(\"/output/path\")\nmodel = LlamaForCausalLM.from_pretrained(\"/output/path\")\n```\n\nNote that executing the script requires enough CPU RAM to host the whole model in float16 precision (even if the biggest versions\ncome in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM). For the 65B model, it's thus 130GB of RAM needed.\n\n- The LLaMA tokenizer is a BPE model based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. \"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\nThis model was contributed by [zphang](https://huggingface.co/zphang) with contributions from [BlackSamorez](https://huggingface.co/BlackSamorez). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox). The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.\n\n\nBased on the original LLaMA model, Meta AI has released some follow-up works:\n\n- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. \ud83c\udf0e\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"710e77e0-7b43-42d7-8a22-6c8e0274eb82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be9a8950-892a-4daa-bc83-ae3d9de68e09\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cb0ca60ff9043520b826d8a931f202e4e38b06d4fdfed2431366ce7814160ffd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d702bd04-353c-4afc-bfb8-8a15e052a03e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f5e831bddeb404184352ad69569fcb49951a541f4cc776139a81b7fb9d6feeec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3192, \"end_char_idx\": 5795, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "ref_doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "chroma:document": "The original code of the authors can be found [here](https://github.com/facebookresearch/llama). The Flax version of the implementation was contributed by [afmck](https://huggingface.co/afmck) with the code in the implementation based on Hugging Face's Flax GPT-Neo.\n\n\nBased on the original LLaMA model, Meta AI has released some follow-up works:\n\n- **Llama2**: Llama2 is an improved version of Llama with some architectural tweaks (Grouped Query Attention), and is pre-trained on 2Trillion tokens. Refer to the documentation of Llama2 which can be found [here](llama2).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/github/bigscience-workshop/petals/blob/main/examples/prompt-tuning-sst2.ipynb#scrollTo=f04ba4d2) on how to use prompt tuning to adapt the LLaMA model for text classification task. \ud83c\udf0e\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [StackLLaMA: A hands-on guide to train LLaMA with RLHF](https://huggingface.co/blog/stackllama#stackllama-a-hands-on-guide-to-train-llama-with-rlhf), a blog post about how to train LLaMA to answer questions on [Stack Exchange](https://stackexchange.com/) with RLHF.\n\n\u2697\ufe0f Optimization\n- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. \ud83c\udf0e \n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the \ud83e\udd17 PEFT library. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the \ud83e\udd17 PEFT library with intuitive UI. \ud83c\udf0e \n- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d702bd04-353c-4afc-bfb8-8a15e052a03e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama", "title": "LLaMA", "tokens": 2025, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d702bd04-353c-4afc-bfb8-8a15e052a03e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"710e77e0-7b43-42d7-8a22-6c8e0274eb82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"30013dab60426b0c6f75250a5fa1216ef98523c9cedcadb2f63b8e0cb3a74204\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"94f1a0b2-5bd2-4ec5-b5e2-a75a15864d1a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama\", \"title\": \"LLaMA\", \"tokens\": 2025, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e8f7de0fc70065103d7fdfac9857a4378e7fc098e65809756fc1752c6399d4a4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4709, \"end_char_idx\": 6754, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "ref_doc_id": "710e77e0-7b43-42d7-8a22-6c8e0274eb82", "chroma:document": "\u2697\ufe0f Optimization\n- A [notebook](https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing) on how to fine-tune LLaMA model using xturing library on GPU which has limited memory. \ud83c\udf0e \n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/github/DominguesM/alpaca-lora-ptbr-7b/blob/main/notebooks/02%20-%20Evaluate.ipynb) on how to run the LLaMA Model using PeftModel from the \ud83e\udd17 PEFT library. \ud83c\udf0e \n- A [notebook](https://colab.research.google.com/drive/1l2GiSSPbajVyp2Nk3CFT4t3uH6-5TiBe?usp=sharing) on how to load a PEFT adapter LLaMA model with LangChain. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- A [notebook](https://colab.research.google.com/github/lxe/simple-llama-finetuner/blob/master/Simple_LLaMA_FineTuner.ipynb#scrollTo=3PM_DilAZD8T) on how to fine-tune LLaMA model using LoRA method via the \ud83e\udd17 PEFT library with intuitive UI. \ud83c\udf0e \n- A [notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/text-generation-open-llama.ipynb) on how to deploy Open-LLaMA model for text generation on Amazon SageMaker. \ud83c\udf0e \n\n## LlamaConfig\n\n[[autodoc]] LlamaConfig\n\n## LlamaTokenizer\n\n[[autodoc]] LlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LlamaTokenizerFast\n\n[[autodoc]] LlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## LlamaModel\n\n[[autodoc]] LlamaModel\n    - forward\n\n## LlamaForCausalLM\n\n[[autodoc]] LlamaForCausalLM\n    - forward\n\n## LlamaForSequenceClassification\n\n[[autodoc]] LlamaForSequenceClassification\n    - forward\n\n## LlamaForQuestionAnswering\n\n[[autodoc]] LlamaForQuestionAnswering\n    - forward\n\n## LlamaForTokenClassification\n\n[[autodoc]] LlamaForTokenClassification\n    - forward\n\n## FlaxLlamaModel\n\n[[autodoc]] FlaxLlamaModel\n    - __call__\n\n## FlaxLlamaForCausalLM\n\n[[autodoc]] FlaxLlamaForCausalLM\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2e19f003-df73-4c34-924c-dfac3db8116a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2e19f003-df73-4c34-924c-dfac3db8116a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26c13974-a5da-4e90-b232-3f28979c827f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"890818b5-2c8f-420b-a1e4-73da3128a75f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b1d4e05a9a2ea26c818f5437642aa636fdb448653924674a6de5fc6d7deffba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3383, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26c13974-a5da-4e90-b232-3f28979c827f", "doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "ref_doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "chroma:document": "# LLaVa-NeXT-Video\n\n## Overview\n\nThe LLaVa-NeXT-Video model was proposed in [LLaVA-NeXT: A Strong Zero-shot Video Understanding Model\n](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/) by Yuanhan Zhang, Bo Li, Haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, Chunyuan Li. LLaVa-NeXT-Video improves upon [LLaVa-NeXT](llava_next) by fine-tuning on a mix if video and image dataset thus increasing the model's performance on videos.\n\n[LLaVA-NeXT](llava_next) surprisingly has strong performance in understanding video content in zero-shot fashion with the AnyRes technique that it uses. The AnyRes technique naturally represents a high-resolution image into multiple images. This technique is naturally generalizable to represent videos because videos can be considered as a set of frames (similar to a set of images in LLaVa-NeXT). The current version of LLaVA-NeXT makes use of AnyRes and trains with supervised fine-tuning (SFT) on top of LLaVA-Next on video data to achieves better video understanding capabilities.The model is a current SOTA among open-source models on [VideoMME bench](https://arxiv.org/abs/2405.21075).\n\n\nThe introduction from the blog is the following:\n\nOn January 30, 2024, we released LLaVA-NeXT, an open-source Large Multimodal Model (LMM) that has been trained exclusively on text-image data. With the proposed AnyRes technique, it boosts capabilities in reasoning, OCR, and world knowledge, demonstrating remarkable performance across a spectrum of image-based multimodal understanding tasks, and even exceeding Gemini-Pro on several image benchmarks, e.g. MMMU and MathVista.\n\n**In today\u2019s exploration, we delve into the performance of LLaVA-NeXT within the realm of video understanding tasks. We reveal that LLaVA-NeXT surprisingly has strong performance in understanding video content. The current version of LLaVA-NeXT for videos has several improvements:\n\n- Zero-shot video representation capabilities with AnyRes: The AnyRes technique naturally represents a high-resolution image into multiple images that a pre-trained VIT is able to digest, and forms them into a concantenated sequence. This technique is naturally generalizable to represent videos (consisting of multiple frames), allowing the image-only-trained LLaVA-Next model to perform surprisingly well on video tasks. Notably, this is the first time that LMMs show strong zero-shot modality transfer ability.\n- Inference with length generalization improves on longer videos. The linear scaling technique enables length generalization, allowing LLaVA-NeXT to effectively handle long-video beyond the limitation of the \"max_token_length\" of the LLM.\n- Strong video understanding ability. (1) LLaVA-Next-Image, which combines the above two techniques, yields superior zero-shot performance than open-source LMMs tuned on videos. (2) LLaVA-Next-Video, further supervised fine-tuning (SFT) LLaVA-Next-Image on video data, achieves better video understanding capabilities compared to LLaVA-Next-Image. (3) LLaVA-Next-Video-DPO, which aligns the model response with AI feedback using direct preference optimization (DPO), showing significant performance boost.\n- Efficient deployment and inference with SGLang. It allows 5x faster inference on video tasks, allowing more scalable serving such as million-level video re-captioning. See instructions in our repo.**"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 890818b5-2c8f-420b-a1e4-73da3128a75f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"890818b5-2c8f-420b-a1e4-73da3128a75f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26c13974-a5da-4e90-b232-3f28979c827f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e19f003-df73-4c34-924c-dfac3db8116a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2e010215656dd890c8c51a164f91ea95373e33320f447f144b0c4683b1d328a3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3daa6017-6b85-49dc-b4ae-694096ff7d61\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2c09721d494f545b6e99cce1c4744f9df6c23e6a2c7840dc5f61df1168ce519\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3386, \"end_char_idx\": 6352, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26c13974-a5da-4e90-b232-3f28979c827f", "doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "ref_doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "chroma:document": "This model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/inference).\n\n## Usage tips\n\n- We advise users to use `padding_side=\"left\"` when computing batched generation as it leads to more accurate results. Simply make sure to call `processor.tokenizer.padding_side = \"left\"` before generating.\n\n<Tip warning={true}>\n\n- Llava-Next uses different number of patches for images and thus has to pad the inputs inside modeling code, aside from the padding done when processing the inputs. The default setting is \"left-padding\" if model is in `eval()` mode, otherwise \"right-padding\".\n\n</Tip>\n\n\n- Note that each checkpoint has been trained with a specific prompt format, depending on which large language model (LLM) was used. You can use tokenizer's `apply_chat_template` to format your prompts correctly. Below is an example of how to do that.\n\nWe will use [LLaVA-NeXT-Video-7B-hf](https://huggingface.co/llava-hf/LLaVA-NeXT-Video-7B-hf) and a conversation history of videos and images. Each content field has to be a list of dicts, as follows:\n\n```python\nfrom transformers import LlavaNextVideoProcessor\n\nprocessor = LlavaNextVideoProcessor.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\")\n\nconversation = [\n    {\n        \"role\": \"system\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            {\"type\": \"image\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your visuals\nprint(text_prompt)\n```\n\n## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode.\n    Returns:\n        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).\n    '''"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3daa6017-6b85-49dc-b4ae-694096ff7d61
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3daa6017-6b85-49dc-b4ae-694096ff7d61\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26c13974-a5da-4e90-b232-3f28979c827f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"890818b5-2c8f-420b-a1e4-73da3128a75f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"69d7cd7238273b8e666e9ed7db395dc2886620408fc64a374edc632e631b7461\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"665798f7-26dc-425d-abfa-d1040a97f264\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"517c920fa1528ae0a23a983e7788761f55a82254c5980e254af75eaa75e746a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4870, \"end_char_idx\": 7859, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26c13974-a5da-4e90-b232-3f28979c827f", "doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "ref_doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "chroma:document": "The assistant gives helpful, detailed, and polite answers to the human's questions.\"},\n            ],\n    },\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"What\u2019s shown in this image?\"},\n            {\"type\": \"image\"},\n            ],\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"This image shows a red stop sign.\"},]\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\ntext_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\n\n# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your visuals\nprint(text_prompt)\n```\n\n## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode.\n    Returns:\n        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).\n    '''\n    frames = []\n    container.seek(0)\n    start_index = indices[0]\n    end_index = indices[-1]\n    for i, frame in enumerate(container.decode(video=0)):\n        if i > end_index:\n            break\n        if i >= start_index and i in indices:\n            frames.append(frame)\n    return np.stack([x.to_ndarray(format=\"rgb24\") for x in frames])\n\n# Load the model in half-precision\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = LlavaNextVideoProcessor.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\")\n\n# Load the video as an np.array, sampling uniformly 8 frames (can sample more for longer videos)\nvideo_path = hf_hub_download(repo_id=\"raushan-testing-hf/videos-test\", filename=\"sample_demo_1.mp4\", repo_type=\"dataset\")\ncontainer = av.open(video_path)\ntotal_frames = container.streams.video[0].frames\nindices = np.arange(0, total_frames, total_frames / 8).astype(int)\nvideo = read_video_pyav(container, indices)\n\nconversation = [\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 665798f7-26dc-425d-abfa-d1040a97f264
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"665798f7-26dc-425d-abfa-d1040a97f264\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26c13974-a5da-4e90-b232-3f28979c827f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3daa6017-6b85-49dc-b4ae-694096ff7d61\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cef73082c08fdf10fc812fa6f050b4d307b6fe0dc6d7087756feaddd79e83442\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4d9f2633-24b3-4403-ad65-1ebdb0998d35\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e71dcaf6b64506d99358b9130c6a60b4d1923cf91d4323e535ff36c65a5ea36\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6357, \"end_char_idx\": 9804, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26c13974-a5da-4e90-b232-3f28979c827f", "doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "ref_doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "chroma:document": "frames = []\n    container.seek(0)\n    start_index = indices[0]\n    end_index = indices[-1]\n    for i, frame in enumerate(container.decode(video=0)):\n        if i > end_index:\n            break\n        if i >= start_index and i in indices:\n            frames.append(frame)\n    return np.stack([x.to_ndarray(format=\"rgb24\") for x in frames])\n\n# Load the model in half-precision\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = LlavaNextVideoProcessor.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\")\n\n# Load the video as an np.array, sampling uniformly 8 frames (can sample more for longer videos)\nvideo_path = hf_hub_download(repo_id=\"raushan-testing-hf/videos-test\", filename=\"sample_demo_1.mp4\", repo_type=\"dataset\")\ncontainer = av.open(video_path)\ntotal_frames = container.streams.video[0].frames\nindices = np.arange(0, total_frames, total_frames / 8).astype(int)\nvideo = read_video_pyav(container, indices)\n\nconversation = [\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\n\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```\n\n\n### Mixed Media Mode\n\nThe model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: \n\n```python\nfrom PIL import Image\nimport requests\n\n# Generate from image and video mixed inputs\n# Load and image and write a new prompt\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nconversation = [\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"How many cats are there in the image?\"},\n            {\"type\": \"image\"},\n            ],\n    },\n    {\n\n        \"role\": \"assistant\",\n        \"content\": [{\"type\": \"text\", \"text\": \"There are two cats\"}],\n    },\n    {\n\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Why is this video funny?\"},\n            {\"type\": \"video\"},\n            ],\n    },\n]\nprompt = processor.apply_chat_template(conversation, add_generation_prompt=True)\ninputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors=\"pt\")\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_length=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes for memory efficiency\n\nThe model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. This allows for efficient deployment on resource-constrained cases. \n\nFirst make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4d9f2633-24b3-4403-ad65-1ebdb0998d35
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llava_next_video", "title": "LLaVa-NeXT-Video", "tokens": 2939, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4d9f2633-24b3-4403-ad65-1ebdb0998d35\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"26c13974-a5da-4e90-b232-3f28979c827f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a98521d3394193a343221d1edaaa06f6449fe023663befe28c2316a8440bbb39\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"665798f7-26dc-425d-abfa-d1040a97f264\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llava_next_video\", \"title\": \"LLaVa-NeXT-Video\", \"tokens\": 2939, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ef688eedae3f33c2a036c91591d4e2f8fb5710b72ec0fbc49a42dc6a09cd8d9c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9807, \"end_char_idx\": 11655, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "26c13974-a5da-4e90-b232-3f28979c827f", "doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "ref_doc_id": "26c13974-a5da-4e90-b232-3f28979c827f", "chroma:document": "```python\nfrom transformers import LlavaNextVideoForConditionalGeneration, LlavaNextVideoProcessor\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\"llava-hf/LLaVA-NeXT-Video-7B-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```\n\n\n### Flash-Attention 2 to speed-up generation\n\nAdditionally, we can greatly speed-up model inference by using [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.\n\nTo load and run a model using Flash Attention-2, simply add `attn_implementation=\"flash_attention_2\"` when loading the model as follows:\n\n```python\nfrom transformers import LlavaNextVideoForConditionalGeneration\n\nmodel = LlavaNextVideoForConditionalGeneration.from_pretrained(\n    \"llava-hf/LLaVA-NeXT-Video-7B-hf\", \n    torch_dtype=torch.float16, \n    attn_implementation=\"flash_attention_2\",\n).to(0)\n```\n\n\n\n## LlavaNextVideoConfig\n\n[[autodoc]] LlavaNextVideoConfig\n\n## LlavaNextVideoProcessor\n\n[[autodoc]] LlavaNextVideoProcessor\n\n## LlavaNextVideoImageProcessor\n\n[[autodoc]] LlavaNextVideoImageProcessor\n\n## LlavaNextVideoForConditionalGeneration\n\n[[autodoc]] LlavaNextVideoForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2356badd-3798-4645-a2b9-826e388591a0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2", "title": "XLSR-Wav2Vec2", "tokens": 728, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2356badd-3798-4645-a2b9-826e388591a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2\", \"title\": \"XLSR-Wav2Vec2\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6b2749a9-7a61-44b2-a8d8-604c2506accb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlsr_wav2vec2\", \"title\": \"XLSR-Wav2Vec2\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"346e0f3442ccd70629e1cdcee7c00d1ee4f4f70000b991c4f6c99b678b5498cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6b2749a9-7a61-44b2-a8d8-604c2506accb", "doc_id": "6b2749a9-7a61-44b2-a8d8-604c2506accb", "ref_doc_id": "6b2749a9-7a61-44b2-a8d8-604c2506accb", "chroma:document": "# XLSR-Wav2Vec2\n\n## Overview\n\nThe XLSR-Wav2Vec2 model was proposed in [Unsupervised Cross-Lingual Representation Learning For Speech Recognition](https://arxiv.org/abs/2006.13979) by Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael\nAuli.\n\nThe abstract from the paper is the following:\n\n*This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw\nwaveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over\nmasked latent speech representations and jointly learns a quantization of the latents shared across languages. The\nresulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly\noutperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction\nof 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to\na comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong\nindividual models. Analysis shows that the latent discrete speech representations are shared across languages with\nincreased sharing for related languages. We hope to catalyze research in low-resource speech understanding by releasing\nXLSR-53, a large model pretrained in 53 languages.*\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).\n\nNote: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- XLSR-Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- XLSR-Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2CTCTokenizer`].\n\n<Tip>\n\nXLSR-Wav2Vec2's architecture is based on the Wav2Vec2 model, so one can refer to [Wav2Vec2's documentation page](wav2vec2).\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0e6fb29f-3da7-401c-b2d8-a36a47558c4c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0e6fb29f-3da7-401c-b2d8-a36a47558c4c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b390b1be-dbd7-4d5c-838d-5389f43e6ab3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"887e9a14cc63016d22a7e940f7f04473dd0e16f4c3884228e28b2f6fc48e5524\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25814a6d-fd8a-45ea-b3e1-a74b8a425575\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7f3234812f55db0630f62818f2dfca439c0b58df9d761d6c19610cccc52b3107\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2883, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "doc_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "ref_doc_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "chroma:document": "# MVP\n\n## Overview\n\nThe MVP model was proposed in [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://arxiv.org/abs/2206.12131) by Tianyi Tang, Junyi Li, Wayne Xin Zhao and Ji-Rong Wen.\n\n\nAccording to the abstract,\n\n- MVP follows a standard Transformer encoder-decoder architecture.\n- MVP is supervised pre-trained using labeled datasets.\n- MVP also has task-specific soft prompts to stimulate the model's capacity in performing a certain task.\n- MVP is specially designed for natural language generation and can be adapted to a wide range of generation tasks, including but not limited to summarization, data-to-text generation, open-ended dialogue system, story generation, question answering, question generation, task-oriented dialogue system, commonsense generation, paraphrase generation, text style transfer, and text simplification. Our model can also be adapted to natural language understanding tasks such as sequence classification and (extractive) question answering.\n\nThis model was contributed by [Tianyi Tang](https://huggingface.co/StevenTang). The detailed information and instructions can be found [here](https://github.com/RUCAIBox/MVP).\n\n## Usage tips\n\n- We have released a series of models [here](https://huggingface.co/models?filter=mvp), including MVP, MVP with task-specific prompts, and multi-task pre-trained variants.\n- If you want to use a model without prompts (standard Transformer), you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp')`.\n- If you want to use a model with task-specific prompts, such as summarization, you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization')`.\n- Our model supports lightweight prompt tuning following [Prefix-tuning](https://arxiv.org/abs/2101.00190) with method `set_lightweight_tuning()`.\n\n## Usage examples\n\nFor summarization, it is an example to use MVP and MVP with summarization-specific prompts.\n\n```python\n>>> from transformers import MvpTokenizer, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizer.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp-summarization\")\n\n>>> inputs = tokenizer(\n...     \"Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Why You Shouldn't Quit Your Job\"]\n\n>>> generated_ids = model_with_prompt.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Don't do it if these are your reasons\"]\n```\n\nFor data-to-text generation, it is an example to use MVP and multi-task pre-trained variants."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 25814a6d-fd8a-45ea-b3e1-a74b8a425575
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"25814a6d-fd8a-45ea-b3e1-a74b8a425575\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b390b1be-dbd7-4d5c-838d-5389f43e6ab3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"887e9a14cc63016d22a7e940f7f04473dd0e16f4c3884228e28b2f6fc48e5524\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e6fb29f-3da7-401c-b2d8-a36a47558c4c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"db4697e91e0e69605a8488288c45eeb16242df0379eced24c631fb15d8412f4b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2630177d-e1d9-4995-aafa-d02f1efeb948\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a633ea8106f4a3a4fbb4c31e5b159e1d3da56e1224fa656f6444362ce601d17\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1188, \"end_char_idx\": 4079, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "doc_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "ref_doc_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "chroma:document": "## Usage tips\n\n- We have released a series of models [here](https://huggingface.co/models?filter=mvp), including MVP, MVP with task-specific prompts, and multi-task pre-trained variants.\n- If you want to use a model without prompts (standard Transformer), you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp')`.\n- If you want to use a model with task-specific prompts, such as summarization, you can load it through `MvpForConditionalGeneration.from_pretrained('RUCAIBox/mvp-summarization')`.\n- Our model supports lightweight prompt tuning following [Prefix-tuning](https://arxiv.org/abs/2101.00190) with method `set_lightweight_tuning()`.\n\n## Usage examples\n\nFor summarization, it is an example to use MVP and MVP with summarization-specific prompts.\n\n```python\n>>> from transformers import MvpTokenizer, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizer.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_prompt = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp-summarization\")\n\n>>> inputs = tokenizer(\n...     \"Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Why You Shouldn't Quit Your Job\"]\n\n>>> generated_ids = model_with_prompt.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Don't do it if these are your reasons\"]\n```\n\nFor data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.\n```python\n>>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizerFast.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mtl-data-to-text\")\n\n>>> inputs = tokenizer(\n...     \"Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']\n\n>>> generated_ids = model_with_mtl.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']\n```\n\nFor lightweight tuning, *i.e.*, fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the [original paper](https://arxiv.org/abs/2101.00190)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2630177d-e1d9-4995-aafa-d02f1efeb948
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mvp", "title": "MVP", "tokens": 1508, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2630177d-e1d9-4995-aafa-d02f1efeb948\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b390b1be-dbd7-4d5c-838d-5389f43e6ab3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"887e9a14cc63016d22a7e940f7f04473dd0e16f4c3884228e28b2f6fc48e5524\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25814a6d-fd8a-45ea-b3e1-a74b8a425575\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mvp\", \"title\": \"MVP\", \"tokens\": 1508, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"24f95573d256242fd3fdd81eadb26ca0280f2ebef7c52e1a4c444bc88773d22e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2430, \"end_char_idx\": 5837, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "doc_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "ref_doc_id": "b390b1be-dbd7-4d5c-838d-5389f43e6ab3", "chroma:document": "\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Why You Shouldn't Quit Your Job\"]\n\n>>> generated_ids = model_with_prompt.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n[\"Don't do it if these are your reasons\"]\n```\n\nFor data-to-text generation, it is an example to use MVP and multi-task pre-trained variants.\n```python\n>>> from transformers import MvpTokenizerFast, MvpForConditionalGeneration\n\n>>> tokenizer = MvpTokenizerFast.from_pretrained(\"RUCAIBox/mvp\")\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\")\n>>> model_with_mtl = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mtl-data-to-text\")\n\n>>> inputs = tokenizer(\n...     \"Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man\",\n...     return_tensors=\"pt\",\n... )\n>>> generated_ids = model.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Stan Lee created the character of Iron Man, a fictional superhero appearing in American comic']\n\n>>> generated_ids = model_with_mtl.generate(**inputs)\n>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)\n['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.']\n```\n\nFor lightweight tuning, *i.e.*, fixing the model and only tuning prompts, you can load MVP with randomly initialized prompts or with task-specific prompts. Our code also supports Prefix-tuning with BART following the [original paper](https://arxiv.org/abs/2101.00190).\n\n```python\n>>> from transformers import MvpForConditionalGeneration\n\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mvp\", use_prompt=True)\n>>> # the number of trainable parameters (full tuning)\n>>> sum(p.numel() for p in model.parameters() if p.requires_grad)\n468116832\n\n>>> # lightweight tuning with randomly initialized prompts\n>>> model.set_lightweight_tuning()\n>>> # the number of trainable parameters (lightweight tuning)\n>>> sum(p.numel() for p in model.parameters() if p.requires_grad)\n61823328\n\n>>> # lightweight tuning with task-specific prompts\n>>> model = MvpForConditionalGeneration.from_pretrained(\"RUCAIBox/mtl-data-to-text\")\n>>> model.set_lightweight_tuning()\n>>> # original lightweight Prefix-tuning\n>>> model = MvpForConditionalGeneration.from_pretrained(\"facebook/bart-large\", use_prompt=True)\n>>> model.set_lightweight_tuning()\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## MvpConfig\n\n[[autodoc]] MvpConfig\n\n## MvpTokenizer\n\n[[autodoc]] MvpTokenizer\n\n## MvpTokenizerFast\n\n[[autodoc]] MvpTokenizerFast\n\n## MvpModel\n\n[[autodoc]] MvpModel\n    - forward\n\n## MvpForConditionalGeneration\n\n[[autodoc]] MvpForConditionalGeneration\n    - forward\n\n## MvpForSequenceClassification\n\n[[autodoc]] MvpForSequenceClassification\n    - forward\n\n## MvpForQuestionAnswering\n\n[[autodoc]] MvpForQuestionAnswering\n    - forward\n\n## MvpForCausalLM\n\n[[autodoc]] MvpForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1164353e-1a01-455d-a18b-65ff1f5c337d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1164353e-1a01-455d-a18b-65ff1f5c337d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a8e6916a-15d7-412d-be87-5457a9047d3c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"68484f21-0320-45fc-85fe-ce27fdafdf7d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8f2925fd6c46cee35196f44f97ef807ca7309c72b612a73365cb45d2fb96a125\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2768, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "ref_doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "chroma:document": "# MBart and MBart-50\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=mbart\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-mbart-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/mbart-large-50-one-to-many-mmt\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n\n## Overview of MBart\n\nThe MBart model was presented in [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan\nGhazvininejad, Mike Lewis, Luke Zettlemoyer.\n\nAccording to the abstract, MBART is a sequence-to-sequence denoising auto-encoder pretrained on large-scale monolingual\ncorpora in many languages using the BART objective. mBART is one of the first methods for pretraining a complete\nsequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only\non the encoder, decoder, or reconstructing parts of the text.\n\nThis model was contributed by [valhalla](https://huggingface.co/valhalla). The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/mbart)\n\n### Training of MBart\n\nMBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for translation task. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nThe regular [`~MBartTokenizer.__call__`] will encode source text format passed as first argument or with the `text`\nkeyword, and target text format passed with the `text_label` keyword argument.\n\n- Supervised training\n\n```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n>>> example_english_phrase = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> expected_translation_romanian = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\n>>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors=\"pt\")\n\n>>> model = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-en-ro\")\n>>> # forward pass\n>>> model(**inputs)\n```\n\n- Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id. The following\n  example shows how to translate English to Romanian using the *facebook/mbart-large-en-ro* model."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 68484f21-0320-45fc-85fe-ce27fdafdf7d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"68484f21-0320-45fc-85fe-ce27fdafdf7d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a8e6916a-15d7-412d-be87-5457a9047d3c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1164353e-1a01-455d-a18b-65ff1f5c337d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"50f4ccd87ca844698bc22bc31462b378be7885029fe5763501b4f8f26590a167\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3adfdae5-184d-4855-9a84-f3f57773c1e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7d6a9301c89809ed9f5de04da89188082c482b01652de5c58c0a0388581f63ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1162, \"end_char_idx\": 4380, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "ref_doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "chroma:document": "The Authors' code can be found [here](https://github.com/pytorch/fairseq/tree/master/examples/mbart)\n\n### Training of MBart\n\nMBart is a multilingual encoder-decoder (sequence-to-sequence) model primarily intended for translation task. As the\nmodel is multilingual it expects the sequences in a different format. A special language id token is added in both the\nsource and target text. The source text format is `X [eos, src_lang_code]` where `X` is the source text. The\ntarget text format is `[tgt_lang_code] X [eos]`. `bos` is never used.\n\nThe regular [`~MBartTokenizer.__call__`] will encode source text format passed as first argument or with the `text`\nkeyword, and target text format passed with the `text_label` keyword argument.\n\n- Supervised training\n\n```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n>>> example_english_phrase = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> expected_translation_romanian = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\n>>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors=\"pt\")\n\n>>> model = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-en-ro\")\n>>> # forward pass\n>>> model(**inputs)\n```\n\n- Generation\n\n  While generating the target text set the `decoder_start_token_id` to the target language id. The following\n  example shows how to translate English to Romanian using the *facebook/mbart-large-en-ro* model.\n\n```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\")\n>>> article = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id[\"ro_RO\"])\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n```\n\n## Overview of MBart-50\n\nMBart-50 was introduced in the [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) paper by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav\nChaudhary, Jiatao Gu, Angela Fan. MBart-50 is created using the original *mbart-large-cc25* checkpoint by extendeding\nits embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50\nlanguages.\n\nAccording to the abstract\n\n*Multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one\ndirection, a pretrained model is finetuned on many directions at the same time. It demonstrates that pretrained models\ncan be extended to incorporate additional languages without loss of performance. Multilingual finetuning improves on\naverage 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while\nimproving 9.3 BLEU on average over bilingual baselines from scratch.*"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3adfdae5-184d-4855-9a84-f3f57773c1e8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3adfdae5-184d-4855-9a84-f3f57773c1e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a8e6916a-15d7-412d-be87-5457a9047d3c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"68484f21-0320-45fc-85fe-ce27fdafdf7d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"94cc98f07015416a55971aa31049e87578efa48682ef655b64290776f43e4065\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8621d0d7-54f6-4f03-9632-9744942b0731\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c7ed3d03c5d33f8f421f6e8976baf8920d60ea504d690d142b74a05b6920824\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2770, \"end_char_idx\": 5870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "ref_doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "chroma:document": "```python\n>>> from transformers import MBartForConditionalGeneration, MBartTokenizer\n\n>>> tokenizer = MBartTokenizer.from_pretrained(\"facebook/mbart-large-en-ro\", src_lang=\"en_XX\")\n>>> article = \"UN Chief Says There Is No Military Solution in Syria\"\n>>> inputs = tokenizer(article, return_tensors=\"pt\")\n>>> translated_tokens = model.generate(**inputs, decoder_start_token_id=tokenizer.lang_code_to_id[\"ro_RO\"])\n>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]\n\"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n```\n\n## Overview of MBart-50\n\nMBart-50 was introduced in the [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/abs/2008.00401) paper by Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav\nChaudhary, Jiatao Gu, Angela Fan. MBart-50 is created using the original *mbart-large-cc25* checkpoint by extendeding\nits embedding layers with randomly initialized vectors for an extra set of 25 language tokens and then pretrained on 50\nlanguages.\n\nAccording to the abstract\n\n*Multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one\ndirection, a pretrained model is finetuned on many directions at the same time. It demonstrates that pretrained models\ncan be extended to incorporate additional languages without loss of performance. Multilingual finetuning improves on\naverage 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while\nimproving 9.3 BLEU on average over bilingual baselines from scratch.*\n\n\n### Training of MBart-50\n\nThe text format for MBart-50 is slightly different from mBART. For MBart-50 the language id token is used as a prefix\nfor both source and target text i.e the text format is `[lang_code] X [eos]`, where `lang_code` is source\nlanguage id for source text and target language id for target text, with `X` being the source or target text\nrespectively.\n\n\nMBart-50 has its own tokenizer [`MBart50Tokenizer`].\n\n-  Supervised training\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n\nsrc_text = \" UN Chief Says There Is No Military Solution in Syria\"\ntgt_text = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\nmodel_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors=\"pt\")\n\nmodel(**model_inputs)  # forward pass\n```\n\n- Generation\n\n  To generate using the mBART-50 multilingual translation models, `eos_token_id` is used as the\n  `decoder_start_token_id` and the target language id is forced as the first generated token. To force the\n  target language id as the first generated token, pass the *forced_bos_token_id* parameter to the *generate* method.\n  The following example shows how to translate between Hindi to French and Arabic to English using the\n  *facebook/mbart-50-large-many-to-many* checkpoint."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8621d0d7-54f6-4f03-9632-9744942b0731
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8621d0d7-54f6-4f03-9632-9744942b0731\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a8e6916a-15d7-412d-be87-5457a9047d3c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3adfdae5-184d-4855-9a84-f3f57773c1e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bb048c9822ed7baa387f47ccaf22bf81238dabce26f1eb13eaffd7cc70d53d4b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c8ffc65-b65e-4825-8e17-4665ee5d4ff9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4346abfd61c417c7387c8b4b974d86dcdcb2d0fe3045e27dc26fb974396f5007\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4379, \"end_char_idx\": 7071, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "ref_doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "chroma:document": "*\n\n\n### Training of MBart-50\n\nThe text format for MBart-50 is slightly different from mBART. For MBart-50 the language id token is used as a prefix\nfor both source and target text i.e the text format is `[lang_code] X [eos]`, where `lang_code` is source\nlanguage id for source text and target language id for target text, with `X` being the source or target text\nrespectively.\n\n\nMBart-50 has its own tokenizer [`MBart50Tokenizer`].\n\n-  Supervised training\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50\", src_lang=\"en_XX\", tgt_lang=\"ro_RO\")\n\nsrc_text = \" UN Chief Says There Is No Military Solution in Syria\"\ntgt_text = \"\u015eeful ONU declar\u0103 c\u0103 nu exist\u0103 o solu\u0163ie militar\u0103 \u00een Siria\"\n\nmodel_inputs = tokenizer(src_text, text_target=tgt_text, return_tensors=\"pt\")\n\nmodel(**model_inputs)  # forward pass\n```\n\n- Generation\n\n  To generate using the mBART-50 multilingual translation models, `eos_token_id` is used as the\n  `decoder_start_token_id` and the target language id is forced as the first generated token. To force the\n  target language id as the first generated token, pass the *forced_bos_token_id* parameter to the *generate* method.\n  The following example shows how to translate between Hindi to French and Arabic to English using the\n  *facebook/mbart-50-large-many-to-many* checkpoint.\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\narticle_hi = \"\u0938\u0902\u092f\u0941\u0915\u094d\u0924 \u0930\u093e\u0937\u094d\u091f\u094d\u0930 \u0915\u0947 \u092a\u094d\u0930\u092e\u0941\u0916 \u0915\u093e \u0915\u0939\u0928\u093e \u0939\u0948 \u0915\u093f \u0938\u0940\u0930\u093f\u092f\u093e \u092e\u0947\u0902 \u0915\u094b\u0908 \u0938\u0948\u0928\u094d\u092f \u0938\u092e\u093e\u0927\u093e\u0928 \u0928\u0939\u0940\u0902 \u0939\u0948\"\narticle_ar = \"\u0627\u0644\u0623\u0645\u064a\u0646 \u0627\u0644\u0639\u0627\u0645 \u0644\u0644\u0623\u0645\u0645 \u0627\u0644\u0645\u062a\u062d\u062f\u0629 \u064a\u0642\u0648\u0644 \u0625\u0646\u0647 \u0644\u0627 \u064a\u0648\u062c\u062f \u062d\u0644 \u0639\u0633\u0643\u0631\u064a \u0641\u064a \u0633\u0648\u0631\u064a\u0627.\"\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\n\n# translate Hindi to French\ntokenizer.src_lang = \"hi_IN\"\nencoded_hi = tokenizer(article_hi, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id[\"fr_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria.\"\n\n# translate Arabic to English\ntokenizer.src_lang = \"ar_AR\"\nencoded_ar = tokenizer(article_ar, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id[\"en_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"The Secretary-General of the United Nations says there is no military solution in Syria.\""}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2c8ffc65-b65e-4825-8e17-4665ee5d4ff9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mbart", "title": "MBart and MBart-50", "tokens": 2452, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2c8ffc65-b65e-4825-8e17-4665ee5d4ff9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a8e6916a-15d7-412d-be87-5457a9047d3c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4640d3b3f56141c584afeed6dc43675d77c1ca80a15de8f902849b14f709db8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8621d0d7-54f6-4f03-9632-9744942b0731\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mbart\", \"title\": \"MBart and MBart-50\", \"tokens\": 2452, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"013f4ccfd221f051de2036473e0a25d792dbc98469991927a6f247b3dffd2692\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5717, \"end_char_idx\": 8786, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "ref_doc_id": "a8e6916a-15d7-412d-be87-5457a9047d3c", "chroma:document": "The following example shows how to translate between Hindi to French and Arabic to English using the\n  *facebook/mbart-50-large-many-to-many* checkpoint.\n\n```python\nfrom transformers import MBartForConditionalGeneration, MBart50TokenizerFast\n\narticle_hi = \"\u0938\u0902\u092f\u0941\u0915\u094d\u0924 \u0930\u093e\u0937\u094d\u091f\u094d\u0930 \u0915\u0947 \u092a\u094d\u0930\u092e\u0941\u0916 \u0915\u093e \u0915\u0939\u0928\u093e \u0939\u0948 \u0915\u093f \u0938\u0940\u0930\u093f\u092f\u093e \u092e\u0947\u0902 \u0915\u094b\u0908 \u0938\u0948\u0928\u094d\u092f \u0938\u092e\u093e\u0927\u093e\u0928 \u0928\u0939\u0940\u0902 \u0939\u0948\"\narticle_ar = \"\u0627\u0644\u0623\u0645\u064a\u0646 \u0627\u0644\u0639\u0627\u0645 \u0644\u0644\u0623\u0645\u0645 \u0627\u0644\u0645\u062a\u062d\u062f\u0629 \u064a\u0642\u0648\u0644 \u0625\u0646\u0647 \u0644\u0627 \u064a\u0648\u062c\u062f \u062d\u0644 \u0639\u0633\u0643\u0631\u064a \u0641\u064a \u0633\u0648\u0631\u064a\u0627.\"\n\nmodel = MBartForConditionalGeneration.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\ntokenizer = MBart50TokenizerFast.from_pretrained(\"facebook/mbart-large-50-many-to-many-mmt\")\n\n# translate Hindi to French\ntokenizer.src_lang = \"hi_IN\"\nencoded_hi = tokenizer(article_hi, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id[\"fr_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria.\"\n\n# translate Arabic to English\ntokenizer.src_lang = \"ar_AR\"\nencoded_ar = tokenizer(article_ar, return_tensors=\"pt\")\ngenerated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id[\"en_XX\"])\ntokenizer.batch_decode(generated_tokens, skip_special_tokens=True)\n# => \"The Secretary-General of the United Nations says there is no military solution in Syria.\"\n```\n\n## Documentation resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## MBartConfig\n\n[[autodoc]] MBartConfig\n\n## MBartTokenizer\n\n[[autodoc]] MBartTokenizer\n    - build_inputs_with_special_tokens\n\n## MBartTokenizerFast\n\n[[autodoc]] MBartTokenizerFast\n\n## MBart50Tokenizer\n\n[[autodoc]] MBart50Tokenizer\n\n## MBart50TokenizerFast\n\n[[autodoc]] MBart50TokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## MBartModel\n\n[[autodoc]] MBartModel\n\n## MBartForConditionalGeneration\n\n[[autodoc]] MBartForConditionalGeneration\n\n## MBartForQuestionAnswering\n\n[[autodoc]] MBartForQuestionAnswering\n\n## MBartForSequenceClassification\n\n[[autodoc]] MBartForSequenceClassification\n\n## MBartForCausalLM\n\n[[autodoc]] MBartForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFMBartModel\n\n[[autodoc]] TFMBartModel\n    - call\n\n## TFMBartForConditionalGeneration\n\n[[autodoc]] TFMBartForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxMBartModel\n\n[[autodoc]] FlaxMBartModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxMBartForConditionalGeneration\n\n[[autodoc]] FlaxMBartForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n## FlaxMBartForSequenceClassification\n\n[[autodoc]] FlaxMBartForSequenceClassification\n    - __call__\n    - encode\n    - decode\n\n## FlaxMBartForQuestionAnswering\n\n[[autodoc]] FlaxMBartForQuestionAnswering\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 18dec7ae-5709-422c-881f-8dd749d4c822
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"18dec7ae-5709-422c-881f-8dd749d4c822\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3b93a359-a5a5-48f4-8c81-5dadd689e583\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4e22a39c16710616f3524a09afb18e0365404d42c4aca02bd883bc532e01407d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3427, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "ref_doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "chroma:document": "# Whisper\n\n## Overview\n\nThe Whisper model was proposed in [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf) by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.\n\nThe abstract from the paper is the following:\n\n*We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.*\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ). The Tensorflow version of this model was contributed by [amyeroberts](https://huggingface.co/amyeroberts).\nThe original code can be found [here](https://github.com/openai/whisper).\n\n## Usage tips\n\n- The model usually performs well without requiring any finetuning.\n- The architecture follows a classic encoder-decoder architecture, which means that it relies on the [`~generation.GenerationMixin.generate`] function for inference.\n- One can use [`WhisperProcessor`] to prepare audio for the model, and decode the predicted ID's back into text.\n\n- To convert the model and the processor, we recommend using the following:\n\n```bash\npython src/transformers/models/whisper/convert_openai_to_hf.py --checkpoint_path \"\" --pytorch_dump_folder_path \"Arthur/whisper-3\" --convert_preprocessor True\n```\nThe script will automatically determine all necessary parameters from the OpenAI checkpoint. A `tiktoken` library needs to be installed\nto perform the conversion of the OpenAI tokenizer to the `tokenizers` version.\n\n## Inference\n\nHere is a step-by-step guide to transcribing an audio sample using a pre-trained Whisper model:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model in Hugging Face format:\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\")\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Generate token ids\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nWhisper is compatible with the following optimisations for both short and long-form generation:\n- [PyTorch Scaled Dot Product Attention (SDPA)](../perf_infer_gpu_one#pytorch-scaled-dot-product-attention): flash attention and memory-efficient attention kernels. Enabled by default for `torch>=2.1.1`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3b93a359-a5a5-48f4-8c81-5dadd689e583\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"18dec7ae-5709-422c-881f-8dd749d4c822\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"88df6a4bd1ddbed60f11986fa864845c71ea287785eb96d75000003c930bb723\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"47eba35f-7b05-45bf-a8c0-f797ff1090fd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d9d8d4c8c3ab09ee309bc50ffd061ccad1082ffc59b7c0e42be5607ee963f5c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1860, \"end_char_idx\": 5293, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "ref_doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "chroma:document": "A `tiktoken` library needs to be installed\nto perform the conversion of the OpenAI tokenizer to the `tokenizers` version.\n\n## Inference\n\nHere is a step-by-step guide to transcribing an audio sample using a pre-trained Whisper model:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model in Hugging Face format:\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\")\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Generate token ids\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nWhisper is compatible with the following optimisations for both short and long-form generation:\n- [PyTorch Scaled Dot Product Attention (SDPA)](../perf_infer_gpu_one#pytorch-scaled-dot-product-attention): flash attention and memory-efficient attention kernels. Enabled by default for `torch>=2.1.1`.\n- [Flash Attention 2](../perf_infer_gpu_one#flashattention-2): improved implementation of flash attention through better parallelism and work partitioning. \n- [torch.compile](../llm_optims#static-kv-cache-and-torchcompile): JIT-compile the forward pass to dispatch to efficient fused kernels.\n\nAs an example, the following codesnippet enables SDPA and `torch.compile` for up to 5x faster inference:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model with SDPA attention\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\", attn_implementation=\"sdpa\")\n\n>>> # Enable static cache and compile the forward pass\n>>> model.generation_config.cache_implementation = \"static\"\n>>> model.forward = torch.compile(model.forward, mode=\"reduce-overhead\", fullgraph=True)\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Compile the forward pass\n>>> for _ in range(2):\n>>>     model.generate(input_features)\n\n>>> # Generate token ids using compiled graph (fast!)\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nFor more details on each optimisation, refer to the documentation linked above."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 47eba35f-7b05-45bf-a8c0-f797ff1090fd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"47eba35f-7b05-45bf-a8c0-f797ff1090fd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3b93a359-a5a5-48f4-8c81-5dadd689e583\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f2dc99c-0a1a-46da-bd83-8107cfe7b9f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a532d81fb42603da715df8dc765c077bc45f4b7826f9a7b24fbf62613d41ceb0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cf2bc405-210d-4b01-b1a2-534d87d2c2d3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"92efc6becf9843ea07cb93a5f1260f8908068d112a667e6e2178ddbf707b2ed6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3585, \"end_char_idx\": 6262, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "ref_doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "chroma:document": "- [torch.compile](../llm_optims#static-kv-cache-and-torchcompile): JIT-compile the forward pass to dispatch to efficient fused kernels.\n\nAs an example, the following codesnippet enables SDPA and `torch.compile` for up to 5x faster inference:\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration\n\n>>> # Select an audio file and read it:\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> audio_sample = ds[0][\"audio\"]\n\n>>> # Load the Whisper model with SDPA attention\n>>> processor = WhisperProcessor.from_pretrained(\"openai/whisper-tiny.en\")\n>>> model = WhisperForConditionalGeneration.from_pretrained(\"openai/whisper-tiny.en\", attn_implementation=\"sdpa\")\n\n>>> # Enable static cache and compile the forward pass\n>>> model.generation_config.cache_implementation = \"static\"\n>>> model.forward = torch.compile(model.forward, mode=\"reduce-overhead\", fullgraph=True)\n\n>>> # Use the model and processor to transcribe the audio:\n>>> input_features = processor(\n...     audio_sample[\"array\"], sampling_rate=audio_sample[\"sampling_rate\"], return_tensors=\"pt\"\n... ).input_features\n\n>>> # Compile the forward pass\n>>> for _ in range(2):\n>>>     model.generate(input_features)\n\n>>> # Generate token ids using compiled graph (fast!)\n>>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nFor more details on each optimisation, refer to the documentation linked above.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Whisper. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Fine-tune Whisper](https://huggingface.co/blog/fine-tune-whisper) on your own dataset for better downstream performance.\n- [Distil-Whisper](https://huggingface.co/distil-whisper): Upto 6x faster, 2x smaller distilled Whisper models for English. We release the [model checkpoints](https://huggingface.co/distil-whisper), and [distillation code](https://github.com/huggingface/distil-whisper).\n- A fork with a script to [convert a Whisper model in Hugging Face format to OpenAI format](https://github.com/zuazo-forks/transformers/blob/convert_hf_to_openai/src/transformers/models/whisper/convert_hf_to_openai.py)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cf2bc405-210d-4b01-b1a2-534d87d2c2d3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/whisper", "title": "Whisper", "tokens": 2044, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cf2bc405-210d-4b01-b1a2-534d87d2c2d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3b93a359-a5a5-48f4-8c81-5dadd689e583\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f3de8d75ccbdfee994c3c40d43fad70aa6c3b3bd7bbd3a00ed5fb10df0eb359c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"47eba35f-7b05-45bf-a8c0-f797ff1090fd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/whisper\", \"title\": \"Whisper\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"96d0c8a8e2f1ce2f0db7a524fca69d17a8c4ed35154c4e7434e4e4038b37f914\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4928, \"end_char_idx\": 8129, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "ref_doc_id": "3b93a359-a5a5-48f4-8c81-5dadd689e583", "chroma:document": ">>> predicted_ids = model.generate(input_features)\n\n>>> # Decode token ids to text\n>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)\n\n>>> transcription[0]\n' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'\n```\n\nFor more details on each optimisation, refer to the documentation linked above.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Whisper. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Fine-tune Whisper](https://huggingface.co/blog/fine-tune-whisper) on your own dataset for better downstream performance.\n- [Distil-Whisper](https://huggingface.co/distil-whisper): Upto 6x faster, 2x smaller distilled Whisper models for English. We release the [model checkpoints](https://huggingface.co/distil-whisper), and [distillation code](https://github.com/huggingface/distil-whisper).\n- A fork with a script to [convert a Whisper model in Hugging Face format to OpenAI format](https://github.com/zuazo-forks/transformers/blob/convert_hf_to_openai/src/transformers/models/whisper/convert_hf_to_openai.py). \ud83c\udf0e\nUsage example:\n```bash\npip install -U openai-whisper\npython convert_hf_to_openai.py \\\n    --checkpoint openai/whisper-tiny \\\n    --whisper_dump_path whisper-tiny-openai.pt\n```\n\n## WhisperConfig\n\n[[autodoc]] WhisperConfig\n\n## WhisperTokenizer\n\n[[autodoc]] WhisperTokenizer\n    - set_prefix_tokens\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n    - batch_decode\n    - decode\n    - basic_normalize\n    - normalize\n\n## WhisperTokenizerFast\n\n[[autodoc]] WhisperTokenizerFast\n    - set_prefix_tokens\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n    - batch_decode\n    - decode\n    - basic_normalize\n    - normalize\n\n## WhisperFeatureExtractor\n\n[[autodoc]] WhisperFeatureExtractor\n    - __call__\n\n## WhisperProcessor\n\n[[autodoc]] WhisperProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n<frameworkcontent>\n<pt>\n\n## WhisperModel\n\n[[autodoc]] WhisperModel\n    - forward\n    - _mask_input_features\n\n## WhisperForConditionalGeneration\n\n[[autodoc]] WhisperForConditionalGeneration\n    - forward\n    - generate\n\n## WhisperForCausalLM\n\n[[autodoc]] WhisperForCausalLM\n    - forward\n\n## WhisperForAudioClassification\n\n[[autodoc]] WhisperForAudioClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFWhisperModel\n\n[[autodoc]] TFWhisperModel\n    - call\n\n## TFWhisperForConditionalGeneration\n\n[[autodoc]] TFWhisperForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxWhisperModel\n\n[[autodoc]] FlaxWhisperModel\n    - __call__\n\n## FlaxWhisperForConditionalGeneration\n\n[[autodoc]] FlaxWhisperForConditionalGeneration\n    - __call__\n\n## FlaxWhisperForAudioClassification\n\n[[autodoc]] FlaxWhisperForAudioClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b8787b14-d2a3-4b00-8e1a-e425fe67dab2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b8787b14-d2a3-4b00-8e1a-e425fe67dab2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f9c3344-ab10-4f0a-b769-b1c778697419\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0542091b-e628-4e97-af39-64766bddac08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dffe5e2e6b910278b431176684075a98e64351ae11c1fbe3912ec5934959ba0a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3255, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "ref_doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "chroma:document": "# LayoutLMv3\n\n## Overview\n\nThe LayoutLMv3 model was proposed in [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://arxiv.org/abs/2204.08387) by Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei.\nLayoutLMv3 simplifies [LayoutLMv2](layoutlmv2) by using patch embeddings (as in [ViT](vit)) instead of leveraging a CNN backbone, and pre-trains the model on 3 objectives: masked language modeling (MLM), masked image modeling (MIM)\nand word-patch alignment (WPA).\n\nThe abstract from the paper is the following:\n\n*Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/layoutlmv3_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> LayoutLMv3 architecture. Taken from the <a href=\"https://arxiv.org/abs/2204.08387\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [chriskoo](https://huggingface.co/chriskoo), [tokec](https://huggingface.co/tokec), and [lre](https://huggingface.co/lre). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/layoutlmv3).\n\n## Usage tips\n\n- In terms of data processing, LayoutLMv3 is identical to its predecessor [LayoutLMv2](layoutlmv2), except that:\n    - images need to be resized and normalized with channels in regular RGB format. LayoutLMv2 on the other hand normalizes the images internally and expects the channels in BGR format.\n    - text is tokenized using byte-pair encoding (BPE), as opposed to WordPiece.\n  Due to these differences in data preprocessing, one can use [`LayoutLMv3Processor`] which internally combines a [`LayoutLMv3ImageProcessor`] (for the image modality) and a [`LayoutLMv3Tokenizer`]/[`LayoutLMv3TokenizerFast`] (for the text modality) to prepare all data for the model.\n- Regarding usage of [`LayoutLMv3Processor`], we refer to the [usage guide](layoutlmv2#usage-layoutlmv2processor) of its predecessor.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv3."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0542091b-e628-4e97-af39-64766bddac08
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0542091b-e628-4e97-af39-64766bddac08\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f9c3344-ab10-4f0a-b769-b1c778697419\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b8787b14-d2a3-4b00-8e1a-e425fe67dab2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"57ae3fdee1c5cb120beb86bd50e8bee629506860197026d05aa2148fcee8e87d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f37e59f9-7588-4deb-bca1-093508856f86\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1aae3006127329e62b24f53eafb906414bce3f9146f9bacb111d67e0a09c6031\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1882, \"end_char_idx\": 4747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "ref_doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2204.08387\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version of this model was added by [chriskoo](https://huggingface.co/chriskoo), [tokec](https://huggingface.co/tokec), and [lre](https://huggingface.co/lre). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/layoutlmv3).\n\n## Usage tips\n\n- In terms of data processing, LayoutLMv3 is identical to its predecessor [LayoutLMv2](layoutlmv2), except that:\n    - images need to be resized and normalized with channels in regular RGB format. LayoutLMv2 on the other hand normalizes the images internally and expects the channels in BGR format.\n    - text is tokenized using byte-pair encoding (BPE), as opposed to WordPiece.\n  Due to these differences in data preprocessing, one can use [`LayoutLMv3Processor`] which internally combines a [`LayoutLMv3ImageProcessor`] (for the image modality) and a [`LayoutLMv3Tokenizer`]/[`LayoutLMv3TokenizerFast`] (for the text modality) to prepare all data for the model.\n- Regarding usage of [`LayoutLMv3Processor`], we refer to the [usage guide](layoutlmv2#usage-layoutlmv2processor) of its predecessor.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv3. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nLayoutLMv3 is nearly identical to LayoutLMv2, so we've also included LayoutLMv2 resources you can adapt for LayoutLMv3 tasks. For these notebooks, take care to use [`LayoutLMv2Processor`] instead when preparing data for the model!\n\n</Tip>\n\n- Demo notebooks for LayoutLMv3 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3).\n- Demo scripts can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3).\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- [`LayoutLMv2ForSequenceClassification`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`LayoutLMv3ForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3) and [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f37e59f9-7588-4deb-bca1-093508856f86
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f37e59f9-7588-4deb-bca1-093508856f86\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f9c3344-ab10-4f0a-b769-b1c778697419\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0542091b-e628-4e97-af39-64766bddac08\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0118d6e526145e53f7bd7e40615ae7e405aa4ef2d175b8807d35c80341fe39df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7c772c3-9430-4ff7-8476-167920c2af46\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2488f3cc1b0f99cdce812989fb0d66b521eebe10dd9ced5cab542bbd25139b0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3128, \"end_char_idx\": 5922, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "ref_doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "chroma:document": "## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LayoutLMv3. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<Tip>\n\nLayoutLMv3 is nearly identical to LayoutLMv2, so we've also included LayoutLMv2 resources you can adapt for LayoutLMv3 tasks. For these notebooks, take care to use [`LayoutLMv2Processor`] instead when preparing data for the model!\n\n</Tip>\n\n- Demo notebooks for LayoutLMv3 can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/LayoutLMv3).\n- Demo scripts can be found [here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3).\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- [`LayoutLMv2ForSequenceClassification`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/RVL-CDIP/Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb).\n- [Text classification task guide](../tasks/sequence_classification)\n\n<PipelineTag pipeline=\"token-classification\"/>\n\n- [`LayoutLMv3ForTokenClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3) and [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb).\n- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb) for how to perform inference with [`LayoutLMv2ForTokenClassification`] and a [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/True_inference_with_LayoutLMv2ForTokenClassification_%2B_Gradio_demo.ipynb) for how to perform inference when no labels are available with [`LayoutLMv2ForTokenClassification`].\n- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) for how to finetune [`LayoutLMv2ForTokenClassification`] with the \ud83e\udd17 Trainer.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`LayoutLMv2ForQuestionAnswering`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c7c772c3-9430-4ff7-8476-167920c2af46
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutlmv3", "title": "LayoutLMv3", "tokens": 2018, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c7c772c3-9430-4ff7-8476-167920c2af46\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6f9c3344-ab10-4f0a-b769-b1c778697419\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"708814cbe26f795e422b191a3d5d9304c1fcda1690b4ef5f69a9fad2853d5873\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f37e59f9-7588-4deb-bca1-093508856f86\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutlmv3\", \"title\": \"LayoutLMv3\", \"tokens\": 2018, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"27ae6aa3482fe78435c1859b23b62486c0b181e0d45c456d33347243b608840c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4748, \"end_char_idx\": 7346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "ref_doc_id": "6f9c3344-ab10-4f0a-b769-b1c778697419", "chroma:document": "- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb) for how to perform inference with [`LayoutLMv2ForTokenClassification`] and a [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/True_inference_with_LayoutLMv2ForTokenClassification_%2B_Gradio_demo.ipynb) for how to perform inference when no labels are available with [`LayoutLMv2ForTokenClassification`].\n- A [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD_using_HuggingFace_Trainer.ipynb) for how to finetune [`LayoutLMv2ForTokenClassification`] with the \ud83e\udd17 Trainer.\n- [Token classification task guide](../tasks/token_classification)\n\n<PipelineTag pipeline=\"question-answering\"/>\n\n- [`LayoutLMv2ForQuestionAnswering`] is supported by this [notebook](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/DocVQA/Fine_tuning_LayoutLMv2ForQuestionAnswering_on_DocVQA.ipynb).\n- [Question answering task guide](../tasks/question_answering)\n\n**Document question answering**\n- [Document question answering task guide](../tasks/document_question_answering)\n\n## LayoutLMv3Config\n\n[[autodoc]] LayoutLMv3Config\n\n## LayoutLMv3FeatureExtractor\n\n[[autodoc]] LayoutLMv3FeatureExtractor\n    - __call__\n\n## LayoutLMv3ImageProcessor\n\n[[autodoc]] LayoutLMv3ImageProcessor\n    - preprocess\n\n## LayoutLMv3Tokenizer\n\n[[autodoc]] LayoutLMv3Tokenizer\n    - __call__\n    - save_vocabulary\n\n## LayoutLMv3TokenizerFast\n\n[[autodoc]] LayoutLMv3TokenizerFast\n    - __call__\n\n## LayoutLMv3Processor\n\n[[autodoc]] LayoutLMv3Processor\n    - __call__\n\n<frameworkcontent>\n<pt>\n\n## LayoutLMv3Model\n\n[[autodoc]] LayoutLMv3Model\n    - forward\n\n## LayoutLMv3ForSequenceClassification\n\n[[autodoc]] LayoutLMv3ForSequenceClassification\n    - forward\n\n## LayoutLMv3ForTokenClassification\n\n[[autodoc]] LayoutLMv3ForTokenClassification\n    - forward\n\n## LayoutLMv3ForQuestionAnswering\n\n[[autodoc]] LayoutLMv3ForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFLayoutLMv3Model\n\n[[autodoc]] TFLayoutLMv3Model\n    - call\n\n## TFLayoutLMv3ForSequenceClassification\n\n[[autodoc]] TFLayoutLMv3ForSequenceClassification\n    - call\n\n## TFLayoutLMv3ForTokenClassification\n\n[[autodoc]] TFLayoutLMv3ForTokenClassification\n    - call\n\n## TFLayoutLMv3ForQuestionAnswering\n\n[[autodoc]] TFLayoutLMv3ForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dbe94015-d003-40bc-978b-3afb357f10a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deformable_detr", "title": "Deformable DETR", "tokens": 934, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dbe94015-d003-40bc-978b-3afb357f10a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deformable_detr\", \"title\": \"Deformable DETR\", \"tokens\": 934, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"94466d58-1b0e-443d-aca7-6ebef02b4837\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deformable_detr\", \"title\": \"Deformable DETR\", \"tokens\": 934, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"856b34e792293fc7baec61f11086da2e16c48653b31dd25f3ab931745405e6f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3267, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "94466d58-1b0e-443d-aca7-6ebef02b4837", "doc_id": "94466d58-1b0e-443d-aca7-6ebef02b4837", "ref_doc_id": "94466d58-1b0e-443d-aca7-6ebef02b4837", "chroma:document": "# Deformable DETR\n\n## Overview\n\nThe Deformable DETR model was proposed in [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://arxiv.org/abs/2010.04159) by Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai.\nDeformable DETR mitigates the slow convergence issues and limited feature spatial resolution of the original [DETR](detr) by leveraging a new deformable attention module which only attends to a small set of key sampling points around a reference.\n\nThe abstract from the paper is the following:\n\n*DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10 times less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/deformable_detr_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Deformable DETR architecture. Taken from the <a href=\"https://arxiv.org/abs/2010.04159\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/fundamentalvision/Deformable-DETR).\n\n## Usage tips\n\n- Training Deformable DETR is equivalent to training the original [DETR](detr) model. See the [resources](#resources) section below for demo notebooks.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Deformable DETR.\n\n<PipelineTag pipeline=\"object-detection\"/>\n\n- Demo notebooks regarding inference + fine-tuning on a custom dataset for [`DeformableDetrForObjectDetection`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Deformable-DETR).\n- Scripts for finetuning [`DeformableDetrForObjectDetection`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/object-detection).\n- See also: [Object detection task guide](../tasks/object_detection).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## DeformableDetrImageProcessor\n\n[[autodoc]] DeformableDetrImageProcessor\n    - preprocess\n    - post_process_object_detection\n\n## DeformableDetrFeatureExtractor\n\n[[autodoc]] DeformableDetrFeatureExtractor\n    - __call__\n    - post_process_object_detection\n\n## DeformableDetrConfig\n\n[[autodoc]] DeformableDetrConfig\n\n## DeformableDetrModel\n\n[[autodoc]] DeformableDetrModel\n    - forward\n\n## DeformableDetrForObjectDetection\n\n[[autodoc]] DeformableDetrForObjectDetection\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6e6a404a-8801-4521-a2ac-b87a5eebae2a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6e6a404a-8801-4521-a2ac-b87a5eebae2a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"66aa280c-4efe-47b6-ba53-58cbaf83975e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2186a2a04b5fb3a1ecdbb8a9d962233c361326beb388a7dbd6c242657ae92abd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d1c85d7-05ff-4337-8b6f-123659dc4663\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7061fb6d69c3beec448cbbb28a29e6878f2b3e380e1f2312c4f5a95746853acf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2492, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "doc_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "ref_doc_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "chroma:document": "# BLIP\n\n## Overview\n\nThe BLIP model was proposed in [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://arxiv.org/abs/2201.12086) by Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi.\n\nBLIP is a model that is able to perform various multi-modal tasks including:\n- Visual Question Answering \n- Image-Text retrieval (Image-text matching)\n- Image Captioning\n\nThe abstract from the paper is the following:\n\n*Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. \nHowever, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*\n\n![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif)\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/salesforce/BLIP).\n\n## Resources\n\n- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset\n\n## BlipConfig\n\n[[autodoc]] BlipConfig\n    - from_text_vision_configs\n\n## BlipTextConfig\n\n[[autodoc]] BlipTextConfig\n\n## BlipVisionConfig\n\n[[autodoc]] BlipVisionConfig\n\n## BlipProcessor\n\n[[autodoc]] BlipProcessor\n\n## BlipImageProcessor\n\n[[autodoc]] BlipImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## BlipModel\n\n`BlipModel` is going to be deprecated in future versions, please use `BlipForConditionalGeneration`, `BlipForImageTextRetrieval` or `BlipForQuestionAnswering` depending on your usecase."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1d1c85d7-05ff-4337-8b6f-123659dc4663
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blip", "title": "BLIP", "tokens": 1084, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1d1c85d7-05ff-4337-8b6f-123659dc4663\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"66aa280c-4efe-47b6-ba53-58cbaf83975e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2186a2a04b5fb3a1ecdbb8a9d962233c361326beb388a7dbd6c242657ae92abd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6e6a404a-8801-4521-a2ac-b87a5eebae2a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blip\", \"title\": \"BLIP\", \"tokens\": 1084, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f1eb5a8e936326b5f5a67e8eff4e5251eba32d7535e1c6853bcc92a3023be554\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1161, \"end_char_idx\": 3472, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "doc_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "ref_doc_id": "66aa280c-4efe-47b6-ba53-58cbaf83975e", "chroma:document": "We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*\n\n![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif)\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada).\nThe original code can be found [here](https://github.com/salesforce/BLIP).\n\n## Resources\n\n- [Jupyter notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_blip.ipynb) on how to fine-tune BLIP for image captioning on a custom dataset\n\n## BlipConfig\n\n[[autodoc]] BlipConfig\n    - from_text_vision_configs\n\n## BlipTextConfig\n\n[[autodoc]] BlipTextConfig\n\n## BlipVisionConfig\n\n[[autodoc]] BlipVisionConfig\n\n## BlipProcessor\n\n[[autodoc]] BlipProcessor\n\n## BlipImageProcessor\n\n[[autodoc]] BlipImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## BlipModel\n\n`BlipModel` is going to be deprecated in future versions, please use `BlipForConditionalGeneration`, `BlipForImageTextRetrieval` or `BlipForQuestionAnswering` depending on your usecase.\n\n[[autodoc]] BlipModel\n    - forward\n    - get_text_features\n    - get_image_features\n\n## BlipTextModel\n\n[[autodoc]] BlipTextModel\n    - forward\n\n## BlipVisionModel\n\n[[autodoc]] BlipVisionModel\n    - forward\n\n## BlipForConditionalGeneration\n\n[[autodoc]] BlipForConditionalGeneration\n    - forward\n\n## BlipForImageTextRetrieval\n\n[[autodoc]] BlipForImageTextRetrieval\n    - forward\n\n## BlipForQuestionAnswering\n\n[[autodoc]] BlipForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFBlipModel\n\n[[autodoc]] TFBlipModel\n    - call\n    - get_text_features\n    - get_image_features\n\n## TFBlipTextModel\n\n[[autodoc]] TFBlipTextModel\n    - call\n\n## TFBlipVisionModel\n\n[[autodoc]] TFBlipVisionModel\n    - call\n\n## TFBlipForConditionalGeneration\n\n[[autodoc]] TFBlipForConditionalGeneration\n    - call\n\n## TFBlipForImageTextRetrieval\n\n[[autodoc]] TFBlipForImageTextRetrieval\n    - call\n\n## TFBlipForQuestionAnswering\n\n[[autodoc]] TFBlipForQuestionAnswering\n    - call\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a9fdaa06-a738-40a5-b907-e11e56ce676e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a9fdaa06-a738-40a5-b907-e11e56ce676e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"76de697c-d2df-4b13-89d7-0a693fcdb3c7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c0009bbca70921622cf3e8b87a72109af7024e115606242d3bb7c90a5f1604c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9e7f4830-22e6-4a53-81d2-f948d9c1fab4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f71057f60e0b274766089ee37f984314fd3ac146e8cb211542163f6dca0c3bbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2719, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "doc_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "ref_doc_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "chroma:document": "# Persimmon\n\n## Overview\n\nThe Persimmon model was created by [ADEPT](https://www.adept.ai/blog/persimmon-8b), and authored by Erich Elsen, Augustus Odena, Maxwell Nye, Sa\u011fnak Ta\u015f\u0131rlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, Arushi Somani.\n\nThe authors introduced Persimmon-8B, a decoder model based on the classic transformers architecture, with query and key normalization. Persimmon-8B is a fully permissively-licensed model with approximately 8 billion parameters, released under the Apache license.  Some of the key attributes of Persimmon-8B are long context size (16K), performance, and capabilities for multimodal extensions.\n\nThe authors showcase their approach to model evaluation, focusing on practical text generation, mirroring how users interact with language models. The work also includes a comparative analysis, pitting Persimmon-8B against other prominent models (MPT 7B Instruct and Llama 2 Base 7B 1-Shot), across various evaluation tasks. The results demonstrate Persimmon-8B's competitive performance, even with limited training data.\n\nIn terms of model details, the work outlines the architecture and training methodology of Persimmon-8B, providing insights into its design choices, sequence length, and dataset composition. The authors present a fast inference code that outperforms traditional implementations through operator fusion and CUDA graph utilization while maintaining code coherence. They express their anticipation of how the community will leverage this contribution to drive innovation, hinting at further upcoming releases as part of an ongoing series of developments.\n\nThis model was contributed by [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/persimmon-ai-labs/adept-inference).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe `Persimmon` models were trained using `bfloat16`, but the original inference uses `float16` The checkpoints uploaded on the hub use `torch_dtype = 'float16'` which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant, unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online) then it will be cast to the default `dtype` of `torch` (becomes `torch.float32`). Users should specify the `torch_dtype` they want, and if they don't it will be `torch.float32`.\n\nFinetuning the model in `float16` is not recommended and known to produce `nan`, as such the model should be fine-tuned in `bfloat16`.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9e7f4830-22e6-4a53-81d2-f948d9c1fab4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/persimmon", "title": "Persimmon", "tokens": 1355, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9e7f4830-22e6-4a53-81d2-f948d9c1fab4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"76de697c-d2df-4b13-89d7-0a693fcdb3c7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8c0009bbca70921622cf3e8b87a72109af7024e115606242d3bb7c90a5f1604c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a9fdaa06-a738-40a5-b907-e11e56ce676e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/persimmon\", \"title\": \"Persimmon\", \"tokens\": 1355, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3c89142ff776d0a8a7ba39fa83d12c343034d00e039a55394235ac01c89dc22\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2722, \"end_char_idx\": 4708, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "doc_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "ref_doc_id": "76de697c-d2df-4b13-89d7-0a693fcdb3c7", "chroma:document": "Tips:\n\n- To convert the model, you need to clone the original repository using `git clone https://github.com/persimmon-ai-labs/adept-inference`, then get the checkpoints:\n\n```bash\ngit clone https://github.com/persimmon-ai-labs/adept-inference\nwget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_base_model_release.tar\ntar -xvf 8b_base_model_release.tar\npython src/transformers/models/persimmon/convert_persimmon_weights_to_hf.py  --input_dir /path/to/downloaded/persimmon/weights/ --output_dir /output/path \\\n    --pt_model_path /path/to/8b_chat_model_release/iter_0001251/mp_rank_00/model_optim_rng.pt\n    --ada_lib_path /path/to/adept-inference\n```\n\nFor the chat model:\n```bash\nwget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar\ntar -xvf 8b_base_model_release.tar\n```\n\nThereafter, models can be loaded via:\n\n```py\nfrom transformers import PersimmonForCausalLM, PersimmonTokenizer\n\nmodel = PersimmonForCausalLM.from_pretrained(\"/output/path\")\ntokenizer = PersimmonTokenizer.from_pretrained(\"/output/path\")\n```\n\n\n- Perismmon uses a `sentencepiece` based tokenizer, with a `Unigram` model. It supports bytefallback, which is only available in `tokenizers==0.14.0` for the fast tokenizer.\nThe `LlamaTokenizer` is used as it is a standard wrapper around sentencepiece. The `chat` template will be updated with the templating functions in a follow up PR!\n\n- The authors suggest to use the following prompt format for the chat mode: `f\"human: {prompt}\\n\\nadept:\"`\n\n\n## PersimmonConfig\n\n[[autodoc]] PersimmonConfig\n\n## PersimmonModel\n\n[[autodoc]] PersimmonModel\n    - forward\n\n## PersimmonForCausalLM\n\n[[autodoc]] PersimmonForCausalLM\n    - forward\n\n## PersimmonForSequenceClassification\n\n[[autodoc]] PersimmonForSequenceClassification\n    - forward\n\n## PersimmonForTokenClassification\n\n[[autodoc]] PersimmonForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0a31ad89-8443-4754-8c17-9034d64df1ed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0a31ad89-8443-4754-8c17-9034d64df1ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c6382b23-3f78-4735-8a73-1c4af58b3449\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c340fd48d6da01aea66faa86738414e5e95cd3647e94d8820eb0862a3175b07\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c602e5c-790e-4401-88ed-1d2a557020ea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf66f3b03383cac5b0b17b826cf800ed42b73ac9bb35089422818eb6dc5dd505\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2167, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "doc_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "ref_doc_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "chroma:document": "# DePlot\n\n## Overview \n\nDePlot was proposed in the paper [DePlot: One-shot visual language reasoning by plot-to-table translation](https://arxiv.org/abs/2212.10505) from Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun.\n\nThe abstract of the paper states the following:\n\n*Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than >28k data points, DePlot+LLM with just one-shot prompting achieves a 24.0% improvement over finetuned SOTA on human-written queries from the task of chart QA.*\n\nDePlot is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).\nDePlot is a Visual Question Answering subset of `Pix2Struct` architecture. It renders the input question on the image and predicts the answer.\n\n## Usage example\n\nCurrently one checkpoint is available for DePlot:\n\n- `google/deplot`: DePlot fine-tuned on ChartQA dataset"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2c602e5c-790e-4401-88ed-1d2a557020ea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/deplot", "title": "DePlot", "tokens": 993, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2c602e5c-790e-4401-88ed-1d2a557020ea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c6382b23-3f78-4735-8a73-1c4af58b3449\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7c340fd48d6da01aea66faa86738414e5e95cd3647e94d8820eb0862a3175b07\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0a31ad89-8443-4754-8c17-9034d64df1ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/deplot\", \"title\": \"DePlot\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"821d0ab478cf69752ce1806e5654d3883f5465bfc4d6b9b74efa6efae4f9e2f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2171, \"end_char_idx\": 3610, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "doc_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "ref_doc_id": "c6382b23-3f78-4735-8a73-1c4af58b3449", "chroma:document": "```python\nfrom transformers import AutoProcessor, Pix2StructForConditionalGeneration\nimport requests\nfrom PIL import Image\n\nmodel = Pix2StructForConditionalGeneration.from_pretrained(\"google/deplot\")\nprocessor = AutoProcessor.from_pretrained(\"google/deplot\")\nurl = \"https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/5090.png\"\nimage = Image.open(requests.get(url, stream=True).raw)\n\ninputs = processor(images=image, text=\"Generate underlying data table of the figure below:\", return_tensors=\"pt\")\npredictions = model.generate(**inputs, max_new_tokens=512)\nprint(processor.decode(predictions[0], skip_special_tokens=True))\n```\n\n## Fine-tuning\n\nTo fine-tune DePlot, refer to the pix2struct [fine-tuning notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb). For `Pix2Struct` models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faster convergence:\n```python\nfrom transformers.optimization import Adafactor, get_cosine_schedule_with_warmup\n\noptimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)\nscheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)\n```\n\n<Tip>\n\nDePlot is a model trained using `Pix2Struct` architecture. For API reference, see [`Pix2Struct` documentation](pix2struct).\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ebcb403f-253c-4113-aec6-5b09c92eed7e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gemma2", "title": "Gemma2", "tokens": 642, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ebcb403f-253c-4113-aec6-5b09c92eed7e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma2\", \"title\": \"Gemma2\", \"tokens\": 642, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"371b66bb-8bf8-4c1d-81a0-a2a20cc13b28\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma2\", \"title\": \"Gemma2\", \"tokens\": 642, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c57ad0e67593dd7b3ee85dbf606c8d01a19dda91ed0b9654f95b4db3949a6934\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1918, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "371b66bb-8bf8-4c1d-81a0-a2a20cc13b28", "doc_id": "371b66bb-8bf8-4c1d-81a0-a2a20cc13b28", "ref_doc_id": "371b66bb-8bf8-4c1d-81a0-a2a20cc13b28", "chroma:document": "# Gemma2\n\n## Overview\n\nThe Gemma2 model was proposed in [Gemma2: Open Models Based on Gemini Technology and Research](https://blog.google/technology/developers/google-gemma-2/) by Gemma2 Team, Google.\nTwo Gemma2 models are released, with parameters sizes of 9 billion (9B) and 27 billion (27B).\n\nThe abstract from the blog post is the following:\n\n*Now we\u2019re officially releasing Gemma 2 to researchers and developers globally. Available in both 9 billion (9B) and 27 billion (27B) parameter sizes, Gemma 2 is higher-performing and more efficient at inference than the first generation, with significant safety advancements built in. In fact, at 27B, it offers competitive alternatives to models more than twice its size, delivering the kind of performance that was only possible with proprietary models as recently as December.*\n\nTips:\n\n- The original checkpoints can be converted using the conversion script `src/transformers/models/Gemma2/convert_Gemma2_weights_to_hf.py` \n\n<Tip warning={true}>\n\n- Gemma2 uses sliding window attention every second layer, which makes it unsuitable for typical kv caching with [`~DynamicCache`] or tuples of tensors. To enable caching in Gemma2 forward call, you must initialize a [`~HybridCache`] instance and pass it as `past_key_values` to the forward call. Note, that you also have to prepare `cache_position` if the `past_key_values` already contains previous keys and values.\n\n</Tip>\n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Pedro Cuenca](https://huggingface.co/pcuenq) and [Tom Arsen]().\n\n\n## Gemma2Config\n\n[[autodoc]] Gemma2Config\n\n## Gemma2Model\n\n[[autodoc]] Gemma2Model\n    - forward\n\n## Gemma2ForCausalLM\n\n[[autodoc]] Gemma2ForCausalLM\n    - forward\n\n## Gemma2ForSequenceClassification\n\n[[autodoc]] Gemma2ForSequenceClassification\n    - forward\n\n## Gemma2ForTokenClassification\n\n[[autodoc]] Gemma2ForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4573eb41-2177-430f-8e96-08d7806d5cb4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4573eb41-2177-430f-8e96-08d7806d5cb4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10331b5d-7fc7-4521-8af3-0df19f465621\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"959aef429e82a136fd83c426f0985cc946b82991b716e40bbf66ed1017be7c3e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c97be010-b686-4efc-b731-616ac1c34823\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"21600adbb10d029becae8e0ef62881c7d29a3ba2dfaba20c69ce1df8e54ecd2a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3519, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "doc_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "ref_doc_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "chroma:document": "# MaskFormer\n\n<Tip>\n\nThis is a recently introduced model so the API hasn't been tested extensively. There may be some bugs or slight\nbreaking changes to fix it in the future. If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title).\n\n</Tip>\n\n## Overview\n\nThe MaskFormer model was proposed in [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://arxiv.org/abs/2107.06278) by Bowen Cheng, Alexander G. Schwing, Alexander Kirillov. MaskFormer addresses semantic segmentation with a mask classification paradigm instead of performing classic pixel-level classification.\n\nThe abstract from the paper is the following:\n\n*Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.*\n\nThe figure below illustrates the architecture of MaskFormer. Taken from the [original paper](https://arxiv.org/abs/2107.06278).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/maskformer_architecture.png\"/>\n\nThis model was contributed by [francesco](https://huggingface.co/francesco). The original code can be found [here](https://github.com/facebookresearch/MaskFormer).\n\n## Usage tips\n\n-  MaskFormer's Transformer decoder is identical to the decoder of [DETR](detr). During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter `use_auxiliary_loss` of [`MaskFormerConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `MaskFormerLoss` class of `modeling_maskformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169).\n- One can use [`MaskFormerImageProcessor`] to prepare images for the model and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~MaskFormerImageProcessor.post_process_semantic_segmentation`] or [`~MaskFormerImageProcessor.post_process_panoptic_segmentation`]."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c97be010-b686-4efc-b731-616ac1c34823
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/maskformer", "title": "MaskFormer", "tokens": 1272, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c97be010-b686-4efc-b731-616ac1c34823\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10331b5d-7fc7-4521-8af3-0df19f465621\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"959aef429e82a136fd83c426f0985cc946b82991b716e40bbf66ed1017be7c3e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4573eb41-2177-430f-8e96-08d7806d5cb4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/maskformer\", \"title\": \"MaskFormer\", \"tokens\": 1272, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5bedfbe5c3d922ecd98e32840fc18e83112fc411491cb5f2d2ff0de18d3de600\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1848, \"end_char_idx\": 5083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "doc_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "ref_doc_id": "10331b5d-7fc7-4521-8af3-0df19f465621", "chroma:document": "Taken from the [original paper](https://arxiv.org/abs/2107.06278).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/maskformer_architecture.png\"/>\n\nThis model was contributed by [francesco](https://huggingface.co/francesco). The original code can be found [here](https://github.com/facebookresearch/MaskFormer).\n\n## Usage tips\n\n-  MaskFormer's Transformer decoder is identical to the decoder of [DETR](detr). During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help the model output the correct number of objects of each class. If you set the parameter `use_auxiliary_loss` of [`MaskFormerConfig`] to `True`, then prediction feedforward neural networks and Hungarian losses are added after each decoder layer (with the FFNs sharing parameters).\n- If you want to train the model in a distributed environment across multiple nodes, then one should update the\n  `get_num_masks` function inside in the `MaskFormerLoss` class of `modeling_maskformer.py`. When training on multiple nodes, this should be\n  set to the average number of target masks across all nodes, as can be seen in the original implementation [here](https://github.com/facebookresearch/MaskFormer/blob/da3e60d85fdeedcb31476b5edd7d328826ce56cc/mask_former/modeling/criterion.py#L169).\n- One can use [`MaskFormerImageProcessor`] to prepare images for the model and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~MaskFormerImageProcessor.post_process_semantic_segmentation`] or [`~MaskFormerImageProcessor.post_process_panoptic_segmentation`]. Both tasks can be solved using [`MaskFormerForInstanceSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\n<PipelineTag pipeline=\"image-segmentation\"/>\n\n- All notebooks that illustrate inference as well as fine-tuning on custom data with MaskFormer can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MaskFormer).\n- Scripts for finetuning [`MaskFormer`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).\n\n## MaskFormer specific outputs\n\n[[autodoc]] models.maskformer.modeling_maskformer.MaskFormerModelOutput\n\n[[autodoc]] models.maskformer.modeling_maskformer.MaskFormerForInstanceSegmentationOutput\n\n## MaskFormerConfig\n\n[[autodoc]] MaskFormerConfig\n\n## MaskFormerImageProcessor\n\n[[autodoc]] MaskFormerImageProcessor\n    - preprocess\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## MaskFormerFeatureExtractor\n\n[[autodoc]] MaskFormerFeatureExtractor\n    - __call__\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation\n\n## MaskFormerModel\n\n[[autodoc]] MaskFormerModel\n    - forward\n\n## MaskFormerForInstanceSegmentation\n\n[[autodoc]] MaskFormerForInstanceSegmentation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8548434a-2c9c-4527-8c75-4e7c1ced2419
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8548434a-2c9c-4527-8c75-4e7c1ced2419\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5af30f47-51a6-45eb-9792-1df28280d1c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1529d2ae10c4b806c09fc27feb4e8171e68a7d065e61569085d01fab102010ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aacbc404-a05d-4334-90f0-40d7cb99476d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbdbd3fbf8d1165877f9b1a4377b2ab5653c71138ab696d44704998c1a27a9c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2491, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "doc_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "ref_doc_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "chroma:document": "# KOSMOS-2\n\n## Overview\n\nThe KOSMOS-2 model was proposed in [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://arxiv.org/abs/2306.14824) by Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei.\n\nKOSMOS-2 is a Transformer-based causal language model and is trained using the next-word prediction task on a web-scale\ndataset of grounded image-text pairs [GRIT](https://huggingface.co/datasets/zzliang/GRIT). The spatial coordinates of\nthe bounding boxes in the dataset are converted to a sequence of location tokens, which are appended to their respective\nentity text spans (for example, `a snowman` followed by `<patch_index_0044><patch_index_0863>`). The data format is\nsimilar to \u201chyperlinks\u201d that connect the object regions in an image to their text span in the corresponding caption.\n\nThe abstract from the paper is the following:\n\n*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href=\"https://arxiv.org/abs/2306.14824\">original paper</a>."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: aacbc404-a05d-4334-90f0-40d7cb99476d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"aacbc404-a05d-4334-90f0-40d7cb99476d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5af30f47-51a6-45eb-9792-1df28280d1c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1529d2ae10c4b806c09fc27feb4e8171e68a7d065e61569085d01fab102010ad\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8548434a-2c9c-4527-8c75-4e7c1ced2419\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd54ade80be2dcdfea779330f953e818640345a1e5aa7004881578f588cd3660\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5599913-cba4-43a5-944e-d41315979cdf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"989c840e4edc78718d1651302ce8a3570ce373969d02f00bc09ebe110edf467d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 706, \"end_char_idx\": 3956, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "doc_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "ref_doc_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "chroma:document": "The data format is\nsimilar to \u201chyperlinks\u201d that connect the object regions in an image to their text span in the corresponding caption.\n\nThe abstract from the paper is the following:\n\n*We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/kosmos_2_overview.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Overview of tasks that KOSMOS-2 can handle. Taken from the <a href=\"https://arxiv.org/abs/2306.14824\">original paper</a>. </small>\n\n## Example\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration\n\n>>> model = Kosmos2ForConditionalGeneration.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n>>> processor = AutoProcessor.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n\n>>> url = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> prompt = \"<grounding> An image of\"\n\n>>> inputs = processor(text=prompt, images=image, return_tensors=\"pt\")\n\n>>> generated_ids = model.generate(\n...     pixel_values=inputs[\"pixel_values\"],\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     image_embeds=None,\n...     image_embeds_position_mask=inputs[\"image_embeds_position_mask\"],\n...     use_cache=True,\n...     max_new_tokens=64,\n... )\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)\n>>> processed_text\n'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'\n\n>>> caption, entities = processor.post_process_generation(generated_text)\n>>> caption\n'An image of a snowman warming himself by a fire.'"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f5599913-cba4-43a5-944e-d41315979cdf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/kosmos-2", "title": "KOSMOS-2", "tokens": 1310, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f5599913-cba4-43a5-944e-d41315979cdf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5af30f47-51a6-45eb-9792-1df28280d1c5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1529d2ae10c4b806c09fc27feb4e8171e68a7d065e61569085d01fab102010ad\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aacbc404-a05d-4334-90f0-40d7cb99476d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/kosmos-2\", \"title\": \"KOSMOS-2\", \"tokens\": 1310, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a0151453595caeb110b469eb9588d2d3ed64dd025e64a4a0775398fe9e2630c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2414, \"end_char_idx\": 4575, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "doc_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "ref_doc_id": "5af30f47-51a6-45eb-9792-1df28280d1c5", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2306.14824\">original paper</a>. </small>\n\n## Example\n\n```python\n>>> from PIL import Image\n>>> import requests\n>>> from transformers import AutoProcessor, Kosmos2ForConditionalGeneration\n\n>>> model = Kosmos2ForConditionalGeneration.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n>>> processor = AutoProcessor.from_pretrained(\"microsoft/kosmos-2-patch14-224\")\n\n>>> url = \"https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg\"\n>>> image = Image.open(requests.get(url, stream=True).raw)\n\n>>> prompt = \"<grounding> An image of\"\n\n>>> inputs = processor(text=prompt, images=image, return_tensors=\"pt\")\n\n>>> generated_ids = model.generate(\n...     pixel_values=inputs[\"pixel_values\"],\n...     input_ids=inputs[\"input_ids\"],\n...     attention_mask=inputs[\"attention_mask\"],\n...     image_embeds=None,\n...     image_embeds_position_mask=inputs[\"image_embeds_position_mask\"],\n...     use_cache=True,\n...     max_new_tokens=64,\n... )\n>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]\n>>> processed_text = processor.post_process_generation(generated_text, cleanup_and_extract=False)\n>>> processed_text\n'<grounding> An image of<phrase> a snowman</phrase><object><patch_index_0044><patch_index_0863></object> warming himself by<phrase> a fire</phrase><object><patch_index_0005><patch_index_0911></object>.'\n\n>>> caption, entities = processor.post_process_generation(generated_text)\n>>> caption\n'An image of a snowman warming himself by a fire.'\n\n>>> entities\n[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]\n```\n\nThis model was contributed by [Yih-Dar SHIEH](https://huggingface.co/ydshieh). The original code can be found [here](https://github.com/microsoft/unilm/tree/master/kosmos-2).\n\n## Kosmos2Config\n\n[[autodoc]] Kosmos2Config\n\n## Kosmos2ImageProcessor\n\n## Kosmos2Processor\n\n[[autodoc]] Kosmos2Processor\n    - __call__\n\n## Kosmos2Model\n\n[[autodoc]] Kosmos2Model\n    - forward\n\n## Kosmos2ForConditionalGeneration\n\n[[autodoc]] Kosmos2ForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9b35abae-7622-4be5-b767-0a5a610df1a4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9b35abae-7622-4be5-b767-0a5a610df1a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2df85ee4-abc8-41d3-84f7-16ca99ae5637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af2b61e7-6097-416b-8cfe-26eb68c62d45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7556100accb5fb4b0ccab26eed5def11201dfcbaccf7da88aa300f1dcd36d9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2939, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "ref_doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "chroma:document": "# Wav2Vec2\n\n## Overview\n\nThe Wav2Vec2 model was proposed in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://arxiv.org/abs/2006.11477) by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.\n\nThe abstract from the paper is the following:\n\n*We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on\ntranscribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks\nthe speech input in the latent space and solves a contrastive task defined over a quantization of the latent\nrepresentations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the\nclean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state\nof the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and\npre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech\nrecognition with limited amounts of labeled data.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).\n\nNote: Meta (FAIR) released a new version of [Wav2Vec2-BERT 2.0](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2-bert) - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: af2b61e7-6097-416b-8cfe-26eb68c62d45
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"af2b61e7-6097-416b-8cfe-26eb68c62d45\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2df85ee4-abc8-41d3-84f7-16ca99ae5637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b35abae-7622-4be5-b767-0a5a610df1a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ea0a6b070eb0da1d4cb317290c369d891d414269fe09d180334f0f2b6a671c07\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8b879c1e-31a9-42b3-b893-77724d1f00a9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7db575ae64e35bb846ad4825283746b02ea946b1ccf23258397cebc31ba4f15a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1429, \"end_char_idx\": 4452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "ref_doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "chroma:document": "We especially recommend using it for fine-tuning tasks, e.g. as per [this guide](https://huggingface.co/blog/fine-tune-w2v2-bert).\n\n## Usage tips\n\n- Wav2Vec2 is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- Wav2Vec2 model was trained using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import Wav2Vec2Model\n\nmodel = Wav2Vec2Model.from_pretrained(\"facebook/wav2vec2-large-960h-lv60-self\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/wav2vec2-large-960h-lv60-self` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: \n\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Wav2Vec2_speedup.png\">\n</div>\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Wav2Vec2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- A notebook on how to [leverage a pretrained Wav2Vec2 model for emotion classification](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8b879c1e-31a9-42b3-b893-77724d1f00a9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8b879c1e-31a9-42b3-b893-77724d1f00a9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2df85ee4-abc8-41d3-84f7-16ca99ae5637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"af2b61e7-6097-416b-8cfe-26eb68c62d45\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a8bb16a2439d0259ae29687d492742fdeb67f2776b4f98f038c15e31333fcad7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8857a361-efae-49b5-ac35-1e453bf7270a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6bcccf13fca0b66112af214c50b34a8ce1dd6fa699a3a6e2143502a6b5dc8435\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2890, \"end_char_idx\": 5854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "ref_doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "chroma:document": "We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import Wav2Vec2Model\n\nmodel = Wav2Vec2Model.from_pretrained(\"facebook/wav2vec2-large-960h-lv60-self\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram comparing the pure inference time between the native implementation in transformers of the `facebook/wav2vec2-large-960h-lv60-self` model and the flash-attention-2 and sdpa (scale-dot-product-attention) versions. . We show the average speedup obtained on the `librispeech_asr` `clean` validation split: \n\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/kamilakesbi/transformers_image_doc/resolve/main/data/Wav2Vec2_speedup.png\">\n</div>\n\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Wav2Vec2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- A notebook on how to [leverage a pretrained Wav2Vec2 model for emotion classification](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb). \ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n- [Audio classification task guide](../tasks/audio_classification)\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- A blog post on [boosting Wav2Vec2 with n-grams in \ud83e\udd17 Transformers](https://huggingface.co/blog/wav2vec2-with-ngram).\n- A blog post on how to [finetune Wav2Vec2 for English ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english).\n- A blog post on [finetuning XLS-R for Multi-Lingual ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2).\n- A notebook on how to [create YouTube captions from any video by transcribing audio with Wav2Vec2](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb). \ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by a notebook on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8857a361-efae-49b5-ac35-1e453bf7270a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8857a361-efae-49b5-ac35-1e453bf7270a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2df85ee4-abc8-41d3-84f7-16ca99ae5637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8b879c1e-31a9-42b3-b893-77724d1f00a9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6c04bb89c3499e550fd8d86c40e77e9f427a122f56068e4a7c97f8964956ba52\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5e074af2-c6db-4abe-a93e-bb778292d8f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb5249c5093421b669e5163ed416f3eea27e3262955d9020962f1ecd2fee5a64\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4453, \"end_char_idx\": 7142, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "ref_doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "chroma:document": "\ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n- [Audio classification task guide](../tasks/audio_classification)\n\n<PipelineTag pipeline=\"automatic-speech-recognition\"/>\n\n- A blog post on [boosting Wav2Vec2 with n-grams in \ud83e\udd17 Transformers](https://huggingface.co/blog/wav2vec2-with-ngram).\n- A blog post on how to [finetune Wav2Vec2 for English ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-wav2vec2-english).\n- A blog post on [finetuning XLS-R for Multi-Lingual ASR with \ud83e\udd17 Transformers](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2).\n- A notebook on how to [create YouTube captions from any video by transcribing audio with Wav2Vec2](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb). \ud83c\udf0e\n- [`Wav2Vec2ForCTC`] is supported by a notebook on [how to finetune a speech recognition model in English](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/speech_recognition.ipynb), and [how to finetune a speech recognition model in any language](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multi_lingual_speech_recognition.ipynb).\n- [Automatic speech recognition task guide](../tasks/asr)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to deploy Wav2Vec2 for [Automatic Speech Recognition with Hugging Face's Transformers & Amazon SageMaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker).\n\n## Wav2Vec2Config\n\n[[autodoc]] Wav2Vec2Config\n\n## Wav2Vec2CTCTokenizer\n\n[[autodoc]] Wav2Vec2CTCTokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n    - set_target_lang\n\n## Wav2Vec2FeatureExtractor\n\n[[autodoc]] Wav2Vec2FeatureExtractor\n    - __call__\n\n## Wav2Vec2Processor\n\n[[autodoc]] Wav2Vec2Processor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Wav2Vec2ProcessorWithLM\n\n[[autodoc]] Wav2Vec2ProcessorWithLM\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n### Decoding multiple audios\n\nIf you are planning to decode multiple batches of audios, you should consider using [`~Wav2Vec2ProcessorWithLM.batch_decode`] and passing an instantiated `multiprocessing.Pool`.\nOtherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5e074af2-c6db-4abe-a93e-bb778292d8f3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5e074af2-c6db-4abe-a93e-bb778292d8f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2df85ee4-abc8-41d3-84f7-16ca99ae5637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8857a361-efae-49b5-ac35-1e453bf7270a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"04679e25c1d70a71e739a8cb72a5f728edaae934988912c5e0a1a4568e41cb0d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b0f510a-0ed3-49e0-b70f-490ed5431744\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7cb29b1fc564efa70e0b6542d82929556ce1233fc00a29d8734cc2e556116205\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5855, \"end_char_idx\": 8546, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "ref_doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "chroma:document": "- [Automatic speech recognition task guide](../tasks/asr)\n\n\ud83d\ude80 Deploy\n\n- A blog post on how to deploy Wav2Vec2 for [Automatic Speech Recognition with Hugging Face's Transformers & Amazon SageMaker](https://www.philschmid.de/automatic-speech-recognition-sagemaker).\n\n## Wav2Vec2Config\n\n[[autodoc]] Wav2Vec2Config\n\n## Wav2Vec2CTCTokenizer\n\n[[autodoc]] Wav2Vec2CTCTokenizer\n    - __call__\n    - save_vocabulary\n    - decode\n    - batch_decode\n    - set_target_lang\n\n## Wav2Vec2FeatureExtractor\n\n[[autodoc]] Wav2Vec2FeatureExtractor\n    - __call__\n\n## Wav2Vec2Processor\n\n[[autodoc]] Wav2Vec2Processor\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Wav2Vec2ProcessorWithLM\n\n[[autodoc]] Wav2Vec2ProcessorWithLM\n    - __call__\n    - pad\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n### Decoding multiple audios\n\nIf you are planning to decode multiple batches of audios, you should consider using [`~Wav2Vec2ProcessorWithLM.batch_decode`] and passing an instantiated `multiprocessing.Pool`.\nOtherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call. See the example below:\n\n```python\n>>> # Let's see how to use a user-managed pool for batch decoding multiple audios\n>>> from multiprocessing import get_context\n>>> from transformers import AutoTokenizer, AutoProcessor, AutoModelForCTC\n>>> from datasets import load_dataset\n>>> import datasets\n>>> import torch\n\n>>> # import model, feature extractor, tokenizer\n>>> model = AutoModelForCTC.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\").to(\"cuda\")\n>>> processor = AutoProcessor.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\")\n\n>>> # load example dataset\n>>> dataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> dataset = dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16_000))\n\n\n>>> def map_to_array(batch):\n...     batch[\"speech\"] = batch[\"audio\"][\"array\"]\n...     return batch\n\n\n>>> # prepare speech data for batch inference\n>>> dataset = dataset.map(map_to_array, remove_columns=[\"audio\"])\n\n\n>>> def map_to_pred(batch, pool):\n...     inputs = processor(batch[\"speech\"], sampling_rate=16_000, padding=True, return_tensors=\"pt\")\n...     inputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\n\n...     with torch.no_grad():\n...         logits = model(**inputs).logits\n\n...     transcription = processor.batch_decode(logits.cpu().numpy(), pool).text\n...     batch[\"transcription\"] = transcription\n...     return batch"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0b0f510a-0ed3-49e0-b70f-490ed5431744
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wav2vec2", "title": "Wav2Vec2", "tokens": 3049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0b0f510a-0ed3-49e0-b70f-490ed5431744\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2df85ee4-abc8-41d3-84f7-16ca99ae5637\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d5e208d757f1471b288b2a870593e14ab6be14c9b48b8761a8fe3f38862ea305\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5e074af2-c6db-4abe-a93e-bb778292d8f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wav2vec2\", \"title\": \"Wav2Vec2\", \"tokens\": 3049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e3db3713dba1c9dabbc5e86495602ff978b3b962b6a6ef290d268fd852ee6256\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6926, \"end_char_idx\": 10652, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "ref_doc_id": "2df85ee4-abc8-41d3-84f7-16ca99ae5637", "chroma:document": "Otherwise, [`~Wav2Vec2ProcessorWithLM.batch_decode`] performance will be slower than calling [`~Wav2Vec2ProcessorWithLM.decode`] for each audio individually, as it internally instantiates a new `Pool` for every call. See the example below:\n\n```python\n>>> # Let's see how to use a user-managed pool for batch decoding multiple audios\n>>> from multiprocessing import get_context\n>>> from transformers import AutoTokenizer, AutoProcessor, AutoModelForCTC\n>>> from datasets import load_dataset\n>>> import datasets\n>>> import torch\n\n>>> # import model, feature extractor, tokenizer\n>>> model = AutoModelForCTC.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\").to(\"cuda\")\n>>> processor = AutoProcessor.from_pretrained(\"patrickvonplaten/wav2vec2-base-100h-with-lm\")\n\n>>> # load example dataset\n>>> dataset = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> dataset = dataset.cast_column(\"audio\", datasets.Audio(sampling_rate=16_000))\n\n\n>>> def map_to_array(batch):\n...     batch[\"speech\"] = batch[\"audio\"][\"array\"]\n...     return batch\n\n\n>>> # prepare speech data for batch inference\n>>> dataset = dataset.map(map_to_array, remove_columns=[\"audio\"])\n\n\n>>> def map_to_pred(batch, pool):\n...     inputs = processor(batch[\"speech\"], sampling_rate=16_000, padding=True, return_tensors=\"pt\")\n...     inputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\n\n...     with torch.no_grad():\n...         logits = model(**inputs).logits\n\n...     transcription = processor.batch_decode(logits.cpu().numpy(), pool).text\n...     batch[\"transcription\"] = transcription\n...     return batch\n\n\n>>> # note: pool should be instantiated *after* `Wav2Vec2ProcessorWithLM`.\n>>> #       otherwise, the LM won't be available to the pool's sub-processes\n>>> # select number of processes and batch_size based on number of CPU cores available and on dataset size\n>>> with get_context(\"fork\").Pool(processes=2) as pool:\n...     result = dataset.map(\n...         map_to_pred, batched=True, batch_size=2, fn_kwargs={\"pool\": pool}, remove_columns=[\"speech\"]\n...     )\n\n>>> result[\"transcription\"][:2]\n['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL', \"NOR IS MISTER COULTER'S MANNER LESS INTERESTING THAN HIS MATTER\"]\n```\n\n## Wav2Vec2 specific outputs\n\n[[autodoc]] models.wav2vec2_with_lm.processing_wav2vec2_with_lm.Wav2Vec2DecoderWithLMOutput\n\n[[autodoc]] models.wav2vec2.modeling_wav2vec2.Wav2Vec2BaseModelOutput\n\n[[autodoc]] models.wav2vec2.modeling_wav2vec2.Wav2Vec2ForPreTrainingOutput\n\n[[autodoc]] models.wav2vec2.modeling_flax_wav2vec2.FlaxWav2Vec2BaseModelOutput\n\n[[autodoc]] models.wav2vec2.modeling_flax_wav2vec2.FlaxWav2Vec2ForPreTrainingOutput\n\n<frameworkcontent>\n<pt>\n\n## Wav2Vec2Model\n\n[[autodoc]] Wav2Vec2Model\n    - forward\n\n## Wav2Vec2ForCTC\n\n[[autodoc]] Wav2Vec2ForCTC\n    - forward\n    - load_adapter\n\n## Wav2Vec2ForSequenceClassification\n\n[[autodoc]] Wav2Vec2ForSequenceClassification\n    - forward\n\n## Wav2Vec2ForAudioFrameClassification\n\n[[autodoc]] Wav2Vec2ForAudioFrameClassification\n    - forward\n\n## Wav2Vec2ForXVector\n\n[[autodoc]] Wav2Vec2ForXVector\n    - forward\n\n## Wav2Vec2ForPreTraining\n\n[[autodoc]] Wav2Vec2ForPreTraining\n    - forward\n\n</pt>\n<tf>\n\n## TFWav2Vec2Model\n\n[[autodoc]] TFWav2Vec2Model\n    - call\n\n## TFWav2Vec2ForSequenceClassification\n\n[[autodoc]] TFWav2Vec2ForSequenceClassification\n    - call\n\n## TFWav2Vec2ForCTC\n\n[[autodoc]] TFWav2Vec2ForCTC\n    - call\n\n</tf>\n<jax>\n\n## FlaxWav2Vec2Model\n\n[[autodoc]] FlaxWav2Vec2Model\n    - __call__\n\n## FlaxWav2Vec2ForCTC\n\n[[autodoc]] FlaxWav2Vec2ForCTC\n    - __call__\n\n## FlaxWav2Vec2ForPreTraining\n\n[[autodoc]] FlaxWav2Vec2ForPreTraining\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 41618e4f-b8b7-4e03-ab10-20d4c47de2fe
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt-sw3", "title": "GPT-Sw3", "tokens": 727, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"41618e4f-b8b7-4e03-ab10-20d4c47de2fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt-sw3\", \"title\": \"GPT-Sw3\", \"tokens\": 727, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ae3a9ea3-7ae7-4c0a-9a1e-542a9db90955\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt-sw3\", \"title\": \"GPT-Sw3\", \"tokens\": 727, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6750a43e5a76ee025437d6599bb92f92bfe77d5345e000ddd08dfa29b628f405\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2216, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ae3a9ea3-7ae7-4c0a-9a1e-542a9db90955", "doc_id": "ae3a9ea3-7ae7-4c0a-9a1e-542a9db90955", "ref_doc_id": "ae3a9ea3-7ae7-4c0a-9a1e-542a9db90955", "chroma:document": "# GPT-Sw3\n\n## Overview\n\nThe GPT-Sw3 model was first proposed in\n[Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish](http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.376.pdf)\nby Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey \u00d6hman,\nFredrik Carlsson, Magnus Sahlgren.\n\nSince that first paper the authors have extended their work and trained new models on their new 1.2TB corpora named The Nordic Pile.\n\nGPT-Sw3 is a collection of large decoder-only pretrained transformer language models that were developed by AI Sweden\nin collaboration with RISE and the WASP WARA for Media and Language. GPT-Sw3 has been trained on a dataset containing\n320B tokens in Swedish, Norwegian, Danish, Icelandic, English, and programming code. The model was pretrained using a\ncausal language modeling (CLM) objective utilizing the NeMo Megatron GPT implementation.\n\nThis model was contributed by [AI Sweden Models](https://huggingface.co/AI-Sweden-Models).\n\n## Usage example\n\n```python\n>>> from transformers import AutoTokenizer, AutoModelForCausalLM\n\n>>> tokenizer = AutoTokenizer.from_pretrained(\"AI-Sweden-Models/gpt-sw3-356m\")\n>>> model = AutoModelForCausalLM.from_pretrained(\"AI-Sweden-Models/gpt-sw3-356m\")\n\n>>> input_ids = tokenizer(\"Tr\u00e4d \u00e4r fina f\u00f6r att\", return_tensors=\"pt\")[\"input_ids\"]\n\n>>> generated_token_ids = model.generate(inputs=input_ids, max_new_tokens=10, do_sample=True)[0]\n\n>>> print(tokenizer.decode(generated_token_ids))\nTr\u00e4d \u00e4r fina f\u00f6r att de \u00e4r f\u00e4rgstarka. Men ibland \u00e4r det fint\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n<Tip>\n\nThe implementation uses the `GPT2Model` coupled with our `GPTSw3Tokenizer`. Refer to [GPT2Model documentation](gpt2) \nfor API reference and examples.  \n\nNote that sentencepiece is required to use our tokenizer and can be installed with `pip install transformers[sentencepiece]` or `pip install sentencepiece`\n\n</Tip>\n\n## GPTSw3Tokenizer\n\n[[autodoc]] GPTSw3Tokenizer\n    - save_vocabulary"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 63dd9c04-699f-4d01-8ae2-b720fb574244
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vivit", "title": "Video Vision Transformer (ViViT)", "tokens": 546, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"63dd9c04-699f-4d01-8ae2-b720fb574244\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vivit\", \"title\": \"Video Vision Transformer (ViViT)\", \"tokens\": 546, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4f1ac533-5083-4fd7-bbce-7b0af2756f3e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vivit\", \"title\": \"Video Vision Transformer (ViViT)\", \"tokens\": 546, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"81b76087ebefb51cf04926f819331492e567ab75140077acea2eeace33b70520\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1878, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4f1ac533-5083-4fd7-bbce-7b0af2756f3e", "doc_id": "4f1ac533-5083-4fd7-bbce-7b0af2756f3e", "ref_doc_id": "4f1ac533-5083-4fd7-bbce-7b0af2756f3e", "chroma:document": "# Video Vision Transformer (ViViT)\n\n## Overview\n\nThe Vivit model was proposed in [ViViT: A Video Vision Transformer](https://arxiv.org/abs/2103.15691) by Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu\u010di\u0107, Cordelia Schmid.\nThe paper proposes one of the first successful pure-transformer based set of models for video understanding.\n\nThe abstract from the paper is the following:\n\n*We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks.*\n\nThis model was contributed by [jegormeister](https://huggingface.co/jegormeister). The original code (written in JAX) can be found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/vivit).\n\n## VivitConfig\n\n[[autodoc]] VivitConfig\n\n## VivitImageProcessor\n\n[[autodoc]] VivitImageProcessor\n    - preprocess\n\n## VivitModel\n\n[[autodoc]] VivitModel\n    - forward\n\n## VivitForVideoClassification\n\n[[autodoc]] transformers.VivitForVideoClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f9533a70-56ef-4c07-bd7f-bc2ceab60b90
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f9533a70-56ef-4c07-bd7f-bc2ceab60b90\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5904e5dc-8db8-42a6-a741-5db8e0675239\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a180558ea7ddc93a3fd1d4c593f79ba78831de05cd2f2a4ef5312691c6ff1545\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3732c6a2-e154-4f85-929b-1785aa596010\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1cb00cc3a9758088f1fa4dca3c0879a6333099dad9a822387e59dde449ef047c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3320, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "doc_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "ref_doc_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "chroma:document": "# ResNet\n\n## Overview\n\nThe ResNet model was proposed in [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Our implementation follows the small changes made by [Nvidia](https://catalog.ngc.nvidia.com/orgs/nvidia/resources/resnet_50_v1_5_for_pytorch), we apply the `stride=2` for downsampling in bottleneck's `3x3` conv and not in the first `1x1`. This is generally known as \"ResNet v1.5\".\n\nResNet introduced residual connections, they allow to train networks with an unseen number of layers (up to 1000). ResNet won the 2015 ILSVRC & COCO competition, one important milestone in deep computer vision.\n\nThe abstract from the paper is the following:\n\n*Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers.\nThe depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.*\n\nThe figure below illustrates the architecture of ResNet. Taken from the [original paper](https://arxiv.org/abs/1512.03385).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/resnet_architecture.png\"/>\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/KaimingHe/deep-residual-networks).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ResNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ResNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3732c6a2-e154-4f85-929b-1785aa596010
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/resnet", "title": "ResNet", "tokens": 1080, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3732c6a2-e154-4f85-929b-1785aa596010\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5904e5dc-8db8-42a6-a741-5db8e0675239\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a180558ea7ddc93a3fd1d4c593f79ba78831de05cd2f2a4ef5312691c6ff1545\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f9533a70-56ef-4c07-bd7f-bc2ceab60b90\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/resnet\", \"title\": \"ResNet\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6dd7c59ca940db96ecfa36eccea84122226414e047e8ad195414c0564e2344e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1590, \"end_char_idx\": 3893, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "doc_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "ref_doc_id": "5904e5dc-8db8-42a6-a741-5db8e0675239", "chroma:document": "The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.*\n\nThe figure below illustrates the architecture of ResNet. Taken from the [original paper](https://arxiv.org/abs/1512.03385).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/resnet_architecture.png\"/>\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The TensorFlow version of this model was added by [amyeroberts](https://huggingface.co/amyeroberts). The original code can be found [here](https://github.com/KaimingHe/deep-residual-networks).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ResNet.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ResNetForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ResNetConfig\n\n[[autodoc]] ResNetConfig\n\n<frameworkcontent>\n<pt>\n\n## ResNetModel\n\n[[autodoc]] ResNetModel\n    - forward\n\n## ResNetForImageClassification\n\n[[autodoc]] ResNetForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFResNetModel\n\n[[autodoc]] TFResNetModel\n    - call\n\n## TFResNetForImageClassification\n\n[[autodoc]] TFResNetForImageClassification\n    - call\n\n</tf>\n<jax>\n\n## FlaxResNetModel\n\n[[autodoc]] FlaxResNetModel\n    - __call__\n\n## FlaxResNetForImageClassification\n\n[[autodoc]] FlaxResNetForImageClassification\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 627f06b8-dee4-4c13-afb4-08cde39aeea6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/van", "title": "VAN", "tokens": 910, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"627f06b8-dee4-4c13-afb4-08cde39aeea6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/van\", \"title\": \"VAN\", \"tokens\": 910, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"380a5858-5faf-4e03-b684-dba2d9ea731c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/van\", \"title\": \"VAN\", \"tokens\": 910, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f375d38bd35bfae59a1ee8570cb707c4b64710d325f5700eac19745f0f390e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3445, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "380a5858-5faf-4e03-b684-dba2d9ea731c", "doc_id": "380a5858-5faf-4e03-b684-dba2d9ea731c", "ref_doc_id": "380a5858-5faf-4e03-b684-dba2d9ea731c", "chroma:document": "# VAN\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe VAN model was proposed in [Visual Attention Network](https://arxiv.org/abs/2202.09741) by Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu.\n\nThis paper introduces a new attention layer based on convolution operations able to capture both local and distant relationships. This is done by combining normal and large kernel convolution layers. The latter uses a dilated convolution to capture distant correlations.\n\nThe abstract from the paper is the following:\n\n*While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel large kernel attention (LKA) module to enable self-adaptive and long-range correlations in self-attention while avoiding the above issues. We further introduce a novel neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN outperforms the state-of-the-art vision transformers and convolutional neural networks with a large margin in extensive experiments, including image classification, object detection, semantic segmentation, instance segmentation, etc. Code is available at [this https URL](https://github.com/Visual-Attention-Network/VAN-Classification).*\n\nTips:\n\n- VAN does not have an embedding layer, thus the `hidden_states` will have a length equal to the number of stages.\n\nThe figure below illustrates the architecture of a Visual Attention Layer. Taken from the [original paper](https://arxiv.org/abs/2202.09741).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/van_architecture.png\"/>\n\nThis model was contributed by [Francesco](https://huggingface.co/Francesco). The original code can be found [here](https://github.com/Visual-Attention-Network/VAN-Classification).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with VAN.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`VanForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## VanConfig\n\n[[autodoc]] VanConfig\n\n## VanModel\n\n[[autodoc]] VanModel\n    - forward\n\n## VanForImageClassification\n\n[[autodoc]] VanForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 12936783-e88f-4b63-941d-14b93d5c7676
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"12936783-e88f-4b63-941d-14b93d5c7676\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8425a653-41ad-4925-a499-a0ce2f55d3d1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b4cb55347c115a5eb94694b83b507f7ba9c4670f671d4f48760b09ecee768c4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eeef13fb-c29a-4e86-90e6-1d32efe44fda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8af9a48db5e31ca8071c07a3de21b1a02f51533548a9c6c829c8fb3dce797b10\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "doc_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "ref_doc_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "chroma:document": "# FlauBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=flaubert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-flaubert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/flaubert_small_cased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe FlauBERT model was proposed in the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le et al. It's a transformer model pretrained using a masked language\nmodeling (MLM) objective (like BERT).\n\nThe abstract from the paper is the following:\n\n*Language models have become a key step to achieve state-of-the art results in many different Natural Language\nProcessing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way\nto pre-train continuous word representations that can be fine-tuned for a downstream task, along with their\ncontextualization at the sentence level. This has been widely demonstrated for English using contextualized\nrepresentations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,\n2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and\nheterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for\nScientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text\nclassification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the\ntime they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation\nprotocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research\ncommunity for further reproducible experiments in French NLP.*\n\nThis model was contributed by [formiel](https://huggingface.co/formiel). The original code can be found [here](https://github.com/getalp/Flaubert).\n\nTips:\n- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: eeef13fb-c29a-4e86-90e6-1d32efe44fda
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/flaubert", "title": "FlauBERT", "tokens": 1184, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"eeef13fb-c29a-4e86-90e6-1d32efe44fda\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8425a653-41ad-4925-a499-a0ce2f55d3d1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b4cb55347c115a5eb94694b83b507f7ba9c4670f671d4f48760b09ecee768c4a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12936783-e88f-4b63-941d-14b93d5c7676\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/flaubert\", \"title\": \"FlauBERT\", \"tokens\": 1184, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a04ac287db6a646c07704d752bf61fcf734e7a330b4111ae8ad9d8b57ff483a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 559, \"end_char_idx\": 3902, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "doc_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "ref_doc_id": "8425a653-41ad-4925-a499-a0ce2f55d3d1", "chroma:document": "It's a transformer model pretrained using a masked language\nmodeling (MLM) objective (like BERT).\n\nThe abstract from the paper is the following:\n\n*Language models have become a key step to achieve state-of-the art results in many different Natural Language\nProcessing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way\nto pre-train continuous word representations that can be fine-tuned for a downstream task, along with their\ncontextualization at the sentence level. This has been widely demonstrated for English using contextualized\nrepresentations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,\n2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and\nheterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for\nScientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text\nclassification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the\ntime they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation\nprotocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research\ncommunity for further reproducible experiments in French NLP.*\n\nThis model was contributed by [formiel](https://huggingface.co/formiel). The original code can be found [here](https://github.com/getalp/Flaubert).\n\nTips:\n- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## FlaubertConfig\n\n[[autodoc]] FlaubertConfig\n\n## FlaubertTokenizer\n\n[[autodoc]] FlaubertTokenizer\n\n<frameworkcontent>\n<pt>\n\n## FlaubertModel\n\n[[autodoc]] FlaubertModel\n    - forward\n\n## FlaubertWithLMHeadModel\n\n[[autodoc]] FlaubertWithLMHeadModel\n    - forward\n\n## FlaubertForSequenceClassification\n\n[[autodoc]] FlaubertForSequenceClassification\n    - forward\n\n## FlaubertForMultipleChoice\n\n[[autodoc]] FlaubertForMultipleChoice\n    - forward\n\n## FlaubertForTokenClassification\n\n[[autodoc]] FlaubertForTokenClassification\n    - forward\n\n## FlaubertForQuestionAnsweringSimple\n\n[[autodoc]] FlaubertForQuestionAnsweringSimple\n    - forward\n\n## FlaubertForQuestionAnswering\n\n[[autodoc]] FlaubertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFFlaubertModel\n\n[[autodoc]] TFFlaubertModel\n    - call\n\n## TFFlaubertWithLMHeadModel\n\n[[autodoc]] TFFlaubertWithLMHeadModel\n    - call\n\n## TFFlaubertForSequenceClassification\n\n[[autodoc]] TFFlaubertForSequenceClassification\n    - call\n\n## TFFlaubertForMultipleChoice\n\n[[autodoc]] TFFlaubertForMultipleChoice\n    - call\n\n## TFFlaubertForTokenClassification\n\n[[autodoc]] TFFlaubertForTokenClassification\n    - call\n\n## TFFlaubertForQuestionAnsweringSimple\n\n[[autodoc]] TFFlaubertForQuestionAnsweringSimple\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8bba8785-0388-4331-96d8-db62d1a7cc1a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8bba8785-0388-4331-96d8-db62d1a7cc1a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c678f14e-6613-40fa-8482-e44865016eec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"29a14a8a-fc0c-400d-a808-70575ed51b61\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52c6c2d35a21e77b21af6968aed82910cc28c1ea417ea194e450e40142b01820\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3338, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c678f14e-6613-40fa-8482-e44865016eec", "doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "ref_doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "chroma:document": "# TVP\n\n## Overview\n\nThe text-visual prompting (TVP) framework was proposed in the paper [Text-Visual Prompting for Efficient 2D Temporal Video Grounding](https://arxiv.org/abs/2303.04995) by Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding.\n\nThe abstract from the paper is the following:\n\n*In this paper, we study the problem of temporal video grounding (TVG), which aims to predict the starting/ending time points of moments described by a text sentence within a long untrimmed video. Benefiting from fine-grained 3D visual features, the TVG techniques have achieved remarkable progress in recent years. However, the high complexity of 3D convolutional neural networks (CNNs) makes extracting dense 3D visual features time-consuming, which calls for intensive memory and computing resources. Towards efficient TVG, we propose a novel text-visual prompting (TVP) framework, which incorporates optimized perturbation patterns (that we call \u2018prompts\u2019) into both visual inputs and textual features of a TVG model. In sharp contrast to 3D CNNs, we show that TVP allows us to effectively co-train vision encoder and language encoder in a 2D TVG model and improves the performance of cross-modal feature fusion using only low-complexity sparse 2D visual features. Further, we propose a Temporal-Distance IoU (TDIoU) loss for efficient learning of TVG. Experiments on two benchmark datasets, Charades-STA and ActivityNet Captions datasets, empirically show that the proposed TVP significantly boosts the performance of 2D TVG (e.g., 9.79% improvement on Charades-STA and 30.77% improvement on ActivityNet Captions) and achieves 5\u00d7 inference acceleration over TVG using 3D visual features.*\n\nThis research addresses temporal video grounding (TVG), which is the process of pinpointing the start and end times of specific events in a long video, as described by a text sentence. Text-visual prompting (TVP), is proposed to enhance TVG. TVP involves integrating specially designed patterns, known as 'prompts', into both the visual (image-based) and textual (word-based) input components of a TVG model. These prompts provide additional spatial-temporal context, improving the model's ability to accurately determine event timings in the video. The approach employs 2D visual inputs in place of 3D ones. Although 3D inputs offer more spatial-temporal detail, they are also more time-consuming to process. The use of 2D inputs with the prompting method aims to provide similar levels of context and accuracy more efficiently.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TVP architecture. Taken from the <a href=\"https://arxiv.org/abs/2303.04995\">original paper.</a> </small>\n\nThis model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).\n\n## Usage tips and examples\n\nPrompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.\n\nTVP consists of a visual encoder and cross-modal encoder."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 29a14a8a-fc0c-400d-a808-70575ed51b61
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"29a14a8a-fc0c-400d-a808-70575ed51b61\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c678f14e-6613-40fa-8482-e44865016eec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8bba8785-0388-4331-96d8-db62d1a7cc1a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8e4d4ef49ef34869cf8191a35ed10b85ea8770b505cecd42c4e3a2e8034ac5a3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"50a66a9c-791e-419f-a512-ba5394880b57\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"04e76286dc9cbeb8bc68dc4421ef5f180059f9b77b56d0abc68d68b3523c4af5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1688, \"end_char_idx\": 4266, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c678f14e-6613-40fa-8482-e44865016eec", "doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "ref_doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "chroma:document": "*\n\nThis research addresses temporal video grounding (TVG), which is the process of pinpointing the start and end times of specific events in a long video, as described by a text sentence. Text-visual prompting (TVP), is proposed to enhance TVG. TVP involves integrating specially designed patterns, known as 'prompts', into both the visual (image-based) and textual (word-based) input components of a TVG model. These prompts provide additional spatial-temporal context, improving the model's ability to accurately determine event timings in the video. The approach employs 2D visual inputs in place of 3D ones. Although 3D inputs offer more spatial-temporal detail, they are also more time-consuming to process. The use of 2D inputs with the prompting method aims to provide similar levels of context and accuracy more efficiently.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TVP architecture. Taken from the <a href=\"https://arxiv.org/abs/2303.04995\">original paper.</a> </small>\n\nThis model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).\n\n## Usage tips and examples\n\nPrompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.\n\nTVP consists of a visual encoder and cross-modal encoder. A universal set of visual prompts and text prompts to be integrated into sampled video frames and textual features, respectively. Specially, a set of different visual prompts are applied to uniformly-sampled frames of one untrimmed video in order.\n\nThe goal of this model is to incorporate trainable prompts into both visual inputs and textual features to temporal video grounding(TVG) problems.\nIn principle, one can apply any visual, cross-modal encoder in the proposed architecture.\n\nThe [`TvpProcessor`] wraps [`BertTokenizer`] and [`TvpImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run temporal video grounding using [`TvpProcessor`] and [`TvpForVideoGrounding`].\n```python\nimport av\nimport cv2\nimport numpy as np\nimport torch\nfrom huggingface_hub import hf_hub_download\nfrom transformers import AutoProcessor, TvpForVideoGrounding"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 50a66a9c-791e-419f-a512-ba5394880b57
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"50a66a9c-791e-419f-a512-ba5394880b57\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c678f14e-6613-40fa-8482-e44865016eec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"29a14a8a-fc0c-400d-a808-70575ed51b61\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f71fe28eba73cd48724aaca12889dc3b59d11ae5030cf7f6aaa0b5d15ea05aae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eb9fe06c-cb75-4d07-962f-02b58b47d309\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f83deaeaed929a8cc079e479a2187704f78b0faa245c7a72eef01c44edc7a2e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2522, \"end_char_idx\": 6125, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c678f14e-6613-40fa-8482-e44865016eec", "doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "ref_doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "chroma:document": "<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/tvp_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> TVP architecture. Taken from the <a href=\"https://arxiv.org/abs/2303.04995\">original paper.</a> </small>\n\nThis model was contributed by [Jiqing Feng](https://huggingface.co/Jiqing). The original code can be found [here](https://github.com/intel/TVP).\n\n## Usage tips and examples\n\nPrompts are optimized perturbation patterns, which would be added to input video frames or text features. Universal set refers to using the same exact set of prompts for any input, this means that these prompts are added consistently to all video frames and text features, regardless of the input's content.\n\nTVP consists of a visual encoder and cross-modal encoder. A universal set of visual prompts and text prompts to be integrated into sampled video frames and textual features, respectively. Specially, a set of different visual prompts are applied to uniformly-sampled frames of one untrimmed video in order.\n\nThe goal of this model is to incorporate trainable prompts into both visual inputs and textual features to temporal video grounding(TVG) problems.\nIn principle, one can apply any visual, cross-modal encoder in the proposed architecture.\n\nThe [`TvpProcessor`] wraps [`BertTokenizer`] and [`TvpImageProcessor`] into a single instance to both\nencode the text and prepare the images respectively.\n\nThe following example shows how to run temporal video grounding using [`TvpProcessor`] and [`TvpForVideoGrounding`].\n```python\nimport av\nimport cv2\nimport numpy as np\nimport torch\nfrom huggingface_hub import hf_hub_download\nfrom transformers import AutoProcessor, TvpForVideoGrounding\n\n\ndef pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps):\n    '''\n    Convert the video from its original fps to the target_fps and decode the video with PyAV decoder.\n    Args:\n        container (container): pyav container.\n        sampling_rate (int): frame sampling rate (interval between two sampled frames).\n        num_frames (int): number of frames to sample.\n        clip_idx (int): if clip_idx is -1, perform random temporal sampling.\n            If clip_idx is larger than -1, uniformly split the video to num_clips\n            clips, and select the clip_idx-th video clip.\n        num_clips (int): overall number of clips to uniformly sample from the given video.\n        target_fps (int): the input video may have different fps, convert it to\n            the target video fps before frame sampling.\n    Returns:\n        frames (tensor): decoded frames from the video. Return None if the no\n            video stream was found.\n        fps (float): the number of frames per second of the video.\n    '''\n    video = container.streams.video[0]\n    fps = float(video.average_rate)\n    clip_size = sampling_rate * num_frames / target_fps * fps\n    delta = max(num_frames - clip_size, 0)\n    start_idx = delta * clip_idx / num_clips\n    end_idx = start_idx + clip_size - 1\n    timebase = video.duration / num_frames\n    video_start_pts = int(start_idx * timebase)\n    video_end_pts = int(end_idx * timebase)\n    seek_offset = max(video_start_pts - 1024, 0)\n    container.seek(seek_offset, any_frame=False, backward=True, stream=video)\n    frames = {}\n    for frame in container.decode(video=0):\n        if frame.pts < video_start_pts:\n            continue\n        frames[frame.pts] = frame\n        if frame.pts > video_end_pts:\n            break\n    frames = [frames[pts] for pts in sorted(frames)]\n    return frames, fps"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: eb9fe06c-cb75-4d07-962f-02b58b47d309
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"eb9fe06c-cb75-4d07-962f-02b58b47d309\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c678f14e-6613-40fa-8482-e44865016eec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"50a66a9c-791e-419f-a512-ba5394880b57\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cdf9c09ed2b4900b3dceb74705ab2b2597387d62ed85fed324c2b39bda96514b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8d844e03-13bc-4660-90e9-be0a7846186e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1d008542f6e1250f373683d4b89bb1e9a5cc15926d23a73c36c8aa343b0a84eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6128, \"end_char_idx\": 9006, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c678f14e-6613-40fa-8482-e44865016eec", "doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "ref_doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "chroma:document": "def decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps):\n    '''\n    Decode the video and perform temporal sampling.\n    Args:\n        container (container): pyav container.\n        sampling_rate (int): frame sampling rate (interval between two sampled frames).\n        num_frames (int): number of frames to sample.\n        clip_idx (int): if clip_idx is -1, perform random temporal sampling.\n            If clip_idx is larger than -1, uniformly split the video to num_clips\n            clips, and select the clip_idx-th video clip.\n        num_clips (int): overall number of clips to uniformly sample from the given video.\n        target_fps (int): the input video may have different fps, convert it to\n            the target video fps before frame sampling.\n    Returns:\n        frames (tensor): decoded frames from the video.\n    '''\n    assert clip_idx >= -2, \"Not a valied clip_idx {}\".format(clip_idx)\n    frames, fps = pyav_decode(container, sampling_rate, num_frames, clip_idx, num_clips, target_fps)\n    clip_size = sampling_rate * num_frames / target_fps * fps\n    index = np.linspace(0, clip_size - 1, num_frames)\n    index = np.clip(index, 0, len(frames) - 1).astype(np.int64)\n    frames = np.array([frames[idx].to_rgb().to_ndarray() for idx in index])\n    frames = frames.transpose(0, 3, 1, 2)\n    return frames\n\n\nfile = hf_hub_download(repo_id=\"Intel/tvp_demo\", filename=\"AK2KG.mp4\", repo_type=\"dataset\")\nmodel = TvpForVideoGrounding.from_pretrained(\"Intel/tvp-base\")\n\ndecoder_kwargs = dict(\n    container=av.open(file, metadata_errors=\"ignore\"),\n    sampling_rate=1,\n    num_frames=model.config.num_frames,\n    clip_idx=0,\n    num_clips=1,\n    target_fps=3,\n)\nraw_sampled_frms = decode(**decoder_kwargs)\n\ntext = \"a person is sitting on a bed.\"\nprocessor = AutoProcessor.from_pretrained(\"Intel/tvp-base\")\nmodel_inputs = processor(\n    text=[text], videos=list(raw_sampled_frms), return_tensors=\"pt\", max_text_length=100#, size=size\n)\n\nmodel_inputs[\"pixel_values\"] = model_inputs[\"pixel_values\"].to(model.dtype)\noutput = model(**model_inputs)\n\ndef get_video_duration(filename):\n    cap = cv2.VideoCapture(filename)\n    if cap.isOpened():\n        rate = cap.get(5)\n        frame_num = cap.get(7)\n        duration = frame_num/rate\n        return duration\n    return -1\n\nduration = get_video_duration(file)\nstart, end = processor.post_process_video_grounding(output.logits, duration)\n\nprint(f\"The time slot of the video corresponding to the text \\\"{text}\\\" is from {start}s to {end}s\")\n```\n\nTips:\n\n- This implementation of TVP uses [`BertTokenizer`] to generate text embeddings and Resnet-50 model to compute visual embeddings.\n- Checkpoints for pre-trained [tvp-base](https://huggingface.co/Intel/tvp-base) is released.\n- Please refer to [Table 2](https://arxiv.org/pdf/2303.04995.pdf) for TVP's performance on Temporal Video Grounding task."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8d844e03-13bc-4660-90e9-be0a7846186e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/tvp", "title": "TVP", "tokens": 2307, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8d844e03-13bc-4660-90e9-be0a7846186e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c678f14e-6613-40fa-8482-e44865016eec\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ffedfc4633315bc3b56a5f08e1d42fb1e444e04116e052131ef925c4f040d801\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eb9fe06c-cb75-4d07-962f-02b58b47d309\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/tvp\", \"title\": \"TVP\", \"tokens\": 2307, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"52789233a75c04d717bcdc3f42ee8c9050b7ae03ae6321de48efd575b2f86bde\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7481, \"end_char_idx\": 9294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c678f14e-6613-40fa-8482-e44865016eec", "doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "ref_doc_id": "c678f14e-6613-40fa-8482-e44865016eec", "chroma:document": "file = hf_hub_download(repo_id=\"Intel/tvp_demo\", filename=\"AK2KG.mp4\", repo_type=\"dataset\")\nmodel = TvpForVideoGrounding.from_pretrained(\"Intel/tvp-base\")\n\ndecoder_kwargs = dict(\n    container=av.open(file, metadata_errors=\"ignore\"),\n    sampling_rate=1,\n    num_frames=model.config.num_frames,\n    clip_idx=0,\n    num_clips=1,\n    target_fps=3,\n)\nraw_sampled_frms = decode(**decoder_kwargs)\n\ntext = \"a person is sitting on a bed.\"\nprocessor = AutoProcessor.from_pretrained(\"Intel/tvp-base\")\nmodel_inputs = processor(\n    text=[text], videos=list(raw_sampled_frms), return_tensors=\"pt\", max_text_length=100#, size=size\n)\n\nmodel_inputs[\"pixel_values\"] = model_inputs[\"pixel_values\"].to(model.dtype)\noutput = model(**model_inputs)\n\ndef get_video_duration(filename):\n    cap = cv2.VideoCapture(filename)\n    if cap.isOpened():\n        rate = cap.get(5)\n        frame_num = cap.get(7)\n        duration = frame_num/rate\n        return duration\n    return -1\n\nduration = get_video_duration(file)\nstart, end = processor.post_process_video_grounding(output.logits, duration)\n\nprint(f\"The time slot of the video corresponding to the text \\\"{text}\\\" is from {start}s to {end}s\")\n```\n\nTips:\n\n- This implementation of TVP uses [`BertTokenizer`] to generate text embeddings and Resnet-50 model to compute visual embeddings.\n- Checkpoints for pre-trained [tvp-base](https://huggingface.co/Intel/tvp-base) is released.\n- Please refer to [Table 2](https://arxiv.org/pdf/2303.04995.pdf) for TVP's performance on Temporal Video Grounding task.\n\n\n## TvpConfig\n\n[[autodoc]] TvpConfig\n\n## TvpImageProcessor\n\n[[autodoc]] TvpImageProcessor\n    - preprocess\n\n## TvpProcessor\n\n[[autodoc]] TvpProcessor\n    - __call__\n\n## TvpModel\n\n[[autodoc]] TvpModel\n    - forward\n\n## TvpForVideoGrounding\n\n[[autodoc]] TvpForVideoGrounding\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a50615ca-492a-40db-b858-f95b9814e1d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a50615ca-492a-40db-b858-f95b9814e1d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec59f234-2721-4b25-9cf8-e9bd7600c5e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0f3d6e2fb79f7d523961ad90a4348c767b483480f117ea156a0fcb3cde7a26c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12d0183b-6923-4d19-8e34-46a0c1dbfe9b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"efdf391577bc1f2dc0c44dbc8bf420acfd67b596e35ddc5964401d114c481485\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "doc_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "ref_doc_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "chroma:document": "# FNet\n\n## Overview\n\nThe FNet model was proposed in [FNet: Mixing Tokens with Fourier Transforms](https://arxiv.org/abs/2105.03824) by\nJames Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. The model replaces the self-attention layer in a BERT\nmodel with a fourier transform which returns only the real parts of the transform. The model is significantly faster\nthan the BERT model because it has fewer parameters and is more memory efficient. The model achieves about 92-97%\naccuracy of BERT counterparts on GLUE benchmark, and trains much faster than the BERT model. The abstract from the\npaper is the following:\n\n*We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the\nself-attention sublayers with simple linear transformations that \"mix\" input tokens. These linear mixers, along with\nstandard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text\nclassification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder\nwith a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE\nbenchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths,\nour FNet model is significantly faster: when compared to the \"efficient\" Transformers on the Long Range Arena\nbenchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all\nsequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint\nand is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models\noutperform Transformer counterparts.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/google-research/google-research/tree/master/f_net).\n\n## Usage tips\n\nThe model was trained without an attention mask as it is based on Fourier Transform. The model was trained with \nmaximum sequence length 512 which includes pad tokens. Hence, it is highly recommended to use the same maximum \nsequence length for fine-tuning and inference."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 12d0183b-6923-4d19-8e34-46a0c1dbfe9b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fnet", "title": "FNet", "tokens": 979, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"12d0183b-6923-4d19-8e34-46a0c1dbfe9b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ec59f234-2721-4b25-9cf8-e9bd7600c5e9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d0f3d6e2fb79f7d523961ad90a4348c767b483480f117ea156a0fcb3cde7a26c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a50615ca-492a-40db-b858-f95b9814e1d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fnet\", \"title\": \"FNet\", \"tokens\": 979, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"65eb9c05e416ac614e89377c6d3c9277561ac24913731ae12cb5c3b301e012d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 454, \"end_char_idx\": 3508, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "doc_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "ref_doc_id": "ec59f234-2721-4b25-9cf8-e9bd7600c5e9", "chroma:document": "The model achieves about 92-97%\naccuracy of BERT counterparts on GLUE benchmark, and trains much faster than the BERT model. The abstract from the\npaper is the following:\n\n*We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the\nself-attention sublayers with simple linear transformations that \"mix\" input tokens. These linear mixers, along with\nstandard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text\nclassification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder\nwith a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE\nbenchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths,\nour FNet model is significantly faster: when compared to the \"efficient\" Transformers on the Long Range Arena\nbenchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all\nsequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint\nand is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models\noutperform Transformer counterparts.*\n\nThis model was contributed by [gchhablani](https://huggingface.co/gchhablani). The original code can be found [here](https://github.com/google-research/google-research/tree/master/f_net).\n\n## Usage tips\n\nThe model was trained without an attention mask as it is based on Fourier Transform. The model was trained with \nmaximum sequence length 512 which includes pad tokens. Hence, it is highly recommended to use the same maximum \nsequence length for fine-tuning and inference.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## FNetConfig\n\n[[autodoc]] FNetConfig\n\n## FNetTokenizer\n\n[[autodoc]] FNetTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## FNetTokenizerFast\n\n[[autodoc]] FNetTokenizerFast\n\n## FNetModel\n\n[[autodoc]] FNetModel\n    - forward\n\n## FNetForPreTraining\n\n[[autodoc]] FNetForPreTraining\n    - forward\n\n## FNetForMaskedLM\n\n[[autodoc]] FNetForMaskedLM\n    - forward\n\n## FNetForNextSentencePrediction\n\n[[autodoc]] FNetForNextSentencePrediction\n    - forward\n\n## FNetForSequenceClassification\n\n[[autodoc]] FNetForSequenceClassification\n    - forward\n\n## FNetForMultipleChoice\n\n[[autodoc]] FNetForMultipleChoice\n    - forward\n\n## FNetForTokenClassification\n\n[[autodoc]] FNetForTokenClassification\n    - forward\n\n## FNetForQuestionAnswering\n\n[[autodoc]] FNetForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f74514fa-00ff-4bcd-a6e8-5f25606186f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/unispeech", "title": "UniSpeech", "tokens": 731, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f74514fa-00ff-4bcd-a6e8-5f25606186f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech\", \"title\": \"UniSpeech\", \"tokens\": 731, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a08dae22-cee2-47ec-89a7-1c7ffa87e8f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech\", \"title\": \"UniSpeech\", \"tokens\": 731, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8577f7a3225609506b83423b7ed57c01159310ecfba559af75d9402fb98b494e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2468, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a08dae22-cee2-47ec-89a7-1c7ffa87e8f4", "doc_id": "a08dae22-cee2-47ec-89a7-1c7ffa87e8f4", "ref_doc_id": "a08dae22-cee2-47ec-89a7-1c7ffa87e8f4", "chroma:document": "# UniSpeech\n\n## Overview\n\nThe UniSpeech model was proposed in [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://arxiv.org/abs/2101.07597) by Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael\nZeng, Xuedong Huang .\n\nThe abstract from the paper is the following:\n\n*In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both\nunlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive\nself-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture\ninformation more correlated with phonetic structures and improve the generalization across languages and domains. We\nevaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The\nresults show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech\nrecognition by a maximum of 13.4% and 17.8% relative phone error rate reductions respectively (averaged over all\ntesting languages). The transferability of UniSpeech is also demonstrated on a domain-shift speech recognition task,\ni.e., a relative word error rate reduction of 6% against the previous approach.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/microsoft/UniSpeech/tree/main/UniSpeech).\n\n## Usage tips\n\n- UniSpeech is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. Please\n  use [`Wav2Vec2Processor`] for the feature extraction.\n- UniSpeech model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2CTCTokenizer`].\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## UniSpeechConfig\n\n[[autodoc]] UniSpeechConfig\n\n## UniSpeech specific outputs\n\n[[autodoc]] models.unispeech.modeling_unispeech.UniSpeechForPreTrainingOutput\n\n## UniSpeechModel\n\n[[autodoc]] UniSpeechModel\n    - forward\n\n## UniSpeechForCTC\n\n[[autodoc]] UniSpeechForCTC\n    - forward\n\n## UniSpeechForSequenceClassification\n\n[[autodoc]] UniSpeechForSequenceClassification\n    - forward\n\n## UniSpeechForPreTraining\n\n[[autodoc]] UniSpeechForPreTraining\n    - forward"}


