# embeddings_queue

created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ab8997a4-6b6b-442b-93f8-56916d21f01c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto/", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ab8997a4-6b6b-442b-93f8-56916d21f01c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7afa274-5d52-4e69-a47a-785ed276650c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92953ab82047ace848d6e82cf3ce41c5cef8da0d6cb71cfea3db6cfb938c4ada\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9bdc1aea-2675-4072-8007-29d16561bb0e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto/\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e1f3f7c3c2f956810313732d62a22768cec7d9d9e39d499976955bceea9c319b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6472, \"end_char_idx\": 7984, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "ref_doc_id": "c7afa274-5d52-4e69-a47a-785ed276650c", "chroma:document": "### AutoModelForAudioClassification\n\n[[autodoc]] AutoModelForAudioClassification\n\n### AutoModelForAudioFrameClassification\n\n[[autodoc]] TFAutoModelForAudioClassification\n\n### TFAutoModelForAudioFrameClassification\n\n[[autodoc]] AutoModelForAudioFrameClassification\n\n### AutoModelForCTC\n\n[[autodoc]] AutoModelForCTC\n\n### AutoModelForSpeechSeq2Seq\n\n[[autodoc]] AutoModelForSpeechSeq2Seq\n\n### TFAutoModelForSpeechSeq2Seq\n\n[[autodoc]] TFAutoModelForSpeechSeq2Seq\n\n### FlaxAutoModelForSpeechSeq2Seq\n\n[[autodoc]] FlaxAutoModelForSpeechSeq2Seq\n\n### AutoModelForAudioXVector\n\n[[autodoc]] AutoModelForAudioXVector\n\n### AutoModelForTextToSpectrogram\n\n[[autodoc]] AutoModelForTextToSpectrogram\n\n### AutoModelForTextToWaveform\n\n[[autodoc]] AutoModelForTextToWaveform\n\n## Multimodal\n\nThe following auto classes are available for the following multimodal tasks.\n\n### AutoModelForTableQuestionAnswering\n\n[[autodoc]] AutoModelForTableQuestionAnswering\n\n### TFAutoModelForTableQuestionAnswering\n\n[[autodoc]] TFAutoModelForTableQuestionAnswering\n\n### AutoModelForDocumentQuestionAnswering\n\n[[autodoc]] AutoModelForDocumentQuestionAnswering\n\n### TFAutoModelForDocumentQuestionAnswering\n\n[[autodoc]] TFAutoModelForDocumentQuestionAnswering\n\n### AutoModelForVisualQuestionAnswering\n\n[[autodoc]] AutoModelForVisualQuestionAnswering\n\n### AutoModelForVision2Seq\n\n[[autodoc]] AutoModelForVision2Seq\n\n### TFAutoModelForVision2Seq\n\n[[autodoc]] TFAutoModelForVision2Seq\n\n### FlaxAutoModelForVision2Seq\n\n[[autodoc]] FlaxAutoModelForVision2Seq"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c7f089e3-b9ae-4ad6-b6e6-a7c977898283
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/instructblip/", "title": "InstructBLIP", "tokens": 835, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c7f089e3-b9ae-4ad6-b6e6-a7c977898283\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/instructblip/\", \"title\": \"InstructBLIP\", \"tokens\": 835, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6082e2cc-dcf7-4b59-89ff-c20d12f73b23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/instructblip/\", \"title\": \"InstructBLIP\", \"tokens\": 835, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"817aee98c1f6903a2a6c027a73477c56941e3ed72634176634d3dab7cf11d5e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6082e2cc-dcf7-4b59-89ff-c20d12f73b23", "doc_id": "6082e2cc-dcf7-4b59-89ff-c20d12f73b23", "ref_doc_id": "6082e2cc-dcf7-4b59-89ff-c20d12f73b23", "chroma:document": "# InstructBLIP\n\n## Overview\n\nThe InstructBLIP model was proposed in [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500) by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.\nInstructBLIP leverages the [BLIP-2](blip2) architecture for visual instruction tuning.\n\nThe abstract from the paper is the following:\n\n*General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/instructblip_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> InstructBLIP architecture. Taken from the <a href=\"https://arxiv.org/abs/2305.06500\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).\n\n## Usage tips\n\nInstructBLIP uses the same architecture as [BLIP-2](blip2) with a tiny but important difference: it also feeds the text prompt (instruction) to the Q-Former.\n\n## InstructBlipConfig\n\n[[autodoc]] InstructBlipConfig\n    - from_vision_qformer_text_configs\n\n## InstructBlipVisionConfig\n\n[[autodoc]] InstructBlipVisionConfig\n\n## InstructBlipQFormerConfig\n\n[[autodoc]] InstructBlipQFormerConfig\n\n## InstructBlipProcessor\n\n[[autodoc]] InstructBlipProcessor\n\n\n## InstructBlipVisionModel\n\n[[autodoc]] InstructBlipVisionModel\n    - forward\n\n## InstructBlipQFormerModel\n\n[[autodoc]] InstructBlipQFormerModel\n    - forward\n\n## InstructBlipForConditionalGeneration\n\n[[autodoc]] InstructBlipForConditionalGeneration\n    - forward\n    - generate"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a95f7c68-c811-4f20-a3e7-a08818b6739f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mpnet/", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a95f7c68-c811-4f20-a3e7-a08818b6739f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet/\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3a238f64-4213-4a81-bc31-ef9554d76c62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet/\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b4cb05c694643d49525cb2dfa95da77bac54448e97701ce234039ccf27eadd1a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"94ce754d-6d53-4ca5-99dc-30fdff0396ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52d5ef87898cc8306b9bf41e59d2a3a99819c8436dd87d1095668ccc9a0ad455\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1959, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3a238f64-4213-4a81-bc31-ef9554d76c62", "doc_id": "3a238f64-4213-4a81-bc31-ef9554d76c62", "ref_doc_id": "3a238f64-4213-4a81-bc31-ef9554d76c62", "chroma:document": "# MPNet\n\n## Overview\n\nThe MPNet model was proposed in [MPNet: Masked and Permuted Pre-training for Language Understanding](https://arxiv.org/abs/2004.09297) by Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu.\n\nMPNet adopts a novel pre-training method, named masked and permuted language modeling, to inherit the advantages of\nmasked language modeling and permuted language modeling for natural language understanding.\n\nThe abstract from the paper is the following:\n\n*BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models.\nSince BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for\npre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and\nthus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel\npre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the\ndependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position\ninformation as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in\nXLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of\ndown-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large\nmargin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g.,\nBERT, XLNet, RoBERTa) under the same model setting.*\n\nThe original code can be found [here](https://github.com/microsoft/MPNet).\n\n## Usage tips\n\nMPNet doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just \nseparate your segments with the separation token `tokenizer.sep_token` (or `[sep]`)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 94ce754d-6d53-4ca5-99dc-30fdff0396ed
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mpnet/", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"94ce754d-6d53-4ca5-99dc-30fdff0396ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet/\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3a238f64-4213-4a81-bc31-ef9554d76c62\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet/\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"b4cb05c694643d49525cb2dfa95da77bac54448e97701ce234039ccf27eadd1a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a95f7c68-c811-4f20-a3e7-a08818b6739f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet/\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"528bb5e5e8d7d69e4aa88341e03813c097082f458828b4c3e5e946d5bc24059e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 217, \"end_char_idx\": 3568, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3a238f64-4213-4a81-bc31-ef9554d76c62", "doc_id": "3a238f64-4213-4a81-bc31-ef9554d76c62", "ref_doc_id": "3a238f64-4213-4a81-bc31-ef9554d76c62", "chroma:document": "MPNet adopts a novel pre-training method, named masked and permuted language modeling, to inherit the advantages of\nmasked language modeling and permuted language modeling for natural language understanding.\n\nThe abstract from the paper is the following:\n\n*BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models.\nSince BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for\npre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and\nthus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel\npre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the\ndependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position\ninformation as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in\nXLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of\ndown-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large\nmargin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g.,\nBERT, XLNet, RoBERTa) under the same model setting.*\n\nThe original code can be found [here](https://github.com/microsoft/MPNet).\n\n## Usage tips\n\nMPNet doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just \nseparate your segments with the separation token `tokenizer.sep_token` (or `[sep]`).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## MPNetConfig\n\n[[autodoc]] MPNetConfig\n\n## MPNetTokenizer\n\n[[autodoc]] MPNetTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## MPNetTokenizerFast\n\n[[autodoc]] MPNetTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## MPNetModel\n\n[[autodoc]] MPNetModel\n    - forward\n\n## MPNetForMaskedLM\n\n[[autodoc]] MPNetForMaskedLM\n    - forward\n\n## MPNetForSequenceClassification\n\n[[autodoc]] MPNetForSequenceClassification\n    - forward\n\n## MPNetForMultipleChoice\n\n[[autodoc]] MPNetForMultipleChoice\n    - forward\n\n## MPNetForTokenClassification\n\n[[autodoc]] MPNetForTokenClassification\n    - forward\n\n## MPNetForQuestionAnswering\n\n[[autodoc]] MPNetForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFMPNetModel\n\n[[autodoc]] TFMPNetModel\n    - call\n\n## TFMPNetForMaskedLM\n\n[[autodoc]] TFMPNetForMaskedLM\n    - call\n\n## TFMPNetForSequenceClassification\n\n[[autodoc]] TFMPNetForSequenceClassification\n    - call\n\n## TFMPNetForMultipleChoice\n\n[[autodoc]] TFMPNetForMultipleChoice\n    - call\n\n## TFMPNetForTokenClassification\n\n[[autodoc]] TFMPNetForTokenClassification\n    - call\n\n## TFMPNetForQuestionAnswering\n\n[[autodoc]] TFMPNetForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cb0fb87a-aaf9-4fe8-9031-b353ba20b342
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnext/", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cb0fb87a-aaf9-4fe8-9031-b353ba20b342\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext/\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab089d5-5169-4397-ac86-a6c95103be81\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext/\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"58793d8311d0accf00c0bca0af776ca6e807e026d79a12feca9c04c0d31ec2ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"77b30fdb-73d0-410f-b4db-69a6201bf65b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4c629fbaf95d990a1d8d6b7a2426a232158392b387882bcc01f1a99fcf3c04d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3244, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab089d5-5169-4397-ac86-a6c95103be81", "doc_id": "dab089d5-5169-4397-ac86-a6c95103be81", "ref_doc_id": "dab089d5-5169-4397-ac86-a6c95103be81", "chroma:document": "# ConvNeXT\n\n## Overview\n\nThe ConvNeXT model was proposed in [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545) by Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie.\nConvNeXT is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, that claims to outperform them.\n\nThe abstract from the paper is the following:\n\n*The \"Roaring 20s\" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model.\nA vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers\n(e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide\nvariety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive\nbiases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually \"modernize\" a standard ResNet toward the design\nof a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models\ndubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy\nand outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ConvNeXT architecture. Taken from the <a href=\"https://arxiv.org/abs/2201.03545\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [ariG23498](https://github.com/ariG23498),\n[gante](https://github.com/gante), and [sayakpaul](https://github.com/sayakpaul) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ConvNeXT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ConvNextForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 77b30fdb-73d0-410f-b4db-69a6201bf65b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnext/", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"77b30fdb-73d0-410f-b4db-69a6201bf65b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext/\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab089d5-5169-4397-ac86-a6c95103be81\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext/\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"58793d8311d0accf00c0bca0af776ca6e807e026d79a12feca9c04c0d31ec2ab\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb0fb87a-aaf9-4fe8-9031-b353ba20b342\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext/\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e76c3dedda92dd0ad1a929061af9d85ab88912fbbec859361f7d8dbf7c0f2d84\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1528, \"end_char_idx\": 3808, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab089d5-5169-4397-ac86-a6c95103be81", "doc_id": "dab089d5-5169-4397-ac86-a6c95103be81", "ref_doc_id": "dab089d5-5169-4397-ac86-a6c95103be81", "chroma:document": "Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy\nand outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ConvNeXT architecture. Taken from the <a href=\"https://arxiv.org/abs/2201.03545\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [ariG23498](https://github.com/ariG23498),\n[gante](https://github.com/gante), and [sayakpaul](https://github.com/sayakpaul) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ConvNeXT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ConvNextForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ConvNextConfig\n\n[[autodoc]] ConvNextConfig\n\n## ConvNextFeatureExtractor\n\n[[autodoc]] ConvNextFeatureExtractor\n\n## ConvNextImageProcessor\n\n[[autodoc]] ConvNextImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## ConvNextModel\n\n[[autodoc]] ConvNextModel\n    - forward\n\n## ConvNextForImageClassification\n\n[[autodoc]] ConvNextForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFConvNextModel\n\n[[autodoc]] TFConvNextModel\n    - call\n\n## TFConvNextForImageClassification\n\n[[autodoc]] TFConvNextForImageClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8d714213-12cd-482f-ab33-46675dad7c8e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer/", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8d714213-12cd-482f-ab33-46675dad7c8e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"faa84358-f68c-49f6-9c18-aa296da38ff4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b3c987491b30eab9215da2400488288fc697783bb4c83b9d84d691b8ce54f26\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c34ca2f4-894f-4f01-9a2e-2a421521b949\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6ef2972ab154af51afa8fa35ef8672b0eedf9815107eb22c4e604abb57e6c1e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3177, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "ref_doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "chroma:document": "# SegFormer\n\n## Overview\n\nThe SegFormer model was proposed in [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://arxiv.org/abs/2105.15203) by Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping\nLuo. The model consists of a hierarchical Transformer encoder and a lightweight all-MLP decode head to achieve great\nresults on image segmentation benchmarks such as ADE20K and Cityscapes.\n\nThe abstract from the paper is the following:\n\n*We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with\nlightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel\nhierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding,\nthereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution\ndiffers from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from\ndifferent layers, and thus combining both local attention and global attention to render powerful representations. We\nshow that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our\napproach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance\nand efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters,\nbeing 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on\nCityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.*\n\nThe figure below illustrates the architecture of SegFormer. Taken from the [original paper](https://arxiv.org/abs/2105.15203).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/segformer_architecture.png\"/>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version\nof the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/NVlabs/SegFormer).\n\n## Usage tips\n\n- SegFormer consists of a hierarchical Transformer encoder, and a lightweight all-MLP decoder head.\n  [`SegformerModel`] is the hierarchical Transformer encoder (which in the paper is also referred to\n  as Mix Transformer or MiT). [`SegformerForSemanticSegmentation`] adds the all-MLP decoder head on\n  top to perform semantic segmentation of images. In addition, there's\n  [`SegformerForImageClassification`] which can be used to - you guessed it - classify images. The\n  authors of SegFormer first pre-trained the Transformer encoder on ImageNet-1k to classify images. Next, they throw\n  away the classification head, and replace it by the all-MLP decode head. Next, they fine-tune the model altogether on\n  ADE20K, Cityscapes and COCO-stuff, which are important benchmarks for semantic segmentation. All checkpoints can be\n  found on the [hub](https://huggingface.co/models?other=segformer)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c34ca2f4-894f-4f01-9a2e-2a421521b949
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer/", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c34ca2f4-894f-4f01-9a2e-2a421521b949\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"faa84358-f68c-49f6-9c18-aa296da38ff4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b3c987491b30eab9215da2400488288fc697783bb4c83b9d84d691b8ce54f26\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8d714213-12cd-482f-ab33-46675dad7c8e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9a5843f9540a7ff111da9913c9993b3abfc7b562cd91953bab7b5b6888385efc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a9289ad-e512-4d53-ba8a-d3bd00616d57\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f336006770b1f40293791d20ff21088930ffa729eb1d39035ab249b25f68baaa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1620, \"end_char_idx\": 4785, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "ref_doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "chroma:document": "Our best model, SegFormer-B5, achieves 84.0% mIoU on\nCityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.*\n\nThe figure below illustrates the architecture of SegFormer. Taken from the [original paper](https://arxiv.org/abs/2105.15203).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/segformer_architecture.png\"/>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version\nof the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/NVlabs/SegFormer).\n\n## Usage tips\n\n- SegFormer consists of a hierarchical Transformer encoder, and a lightweight all-MLP decoder head.\n  [`SegformerModel`] is the hierarchical Transformer encoder (which in the paper is also referred to\n  as Mix Transformer or MiT). [`SegformerForSemanticSegmentation`] adds the all-MLP decoder head on\n  top to perform semantic segmentation of images. In addition, there's\n  [`SegformerForImageClassification`] which can be used to - you guessed it - classify images. The\n  authors of SegFormer first pre-trained the Transformer encoder on ImageNet-1k to classify images. Next, they throw\n  away the classification head, and replace it by the all-MLP decode head. Next, they fine-tune the model altogether on\n  ADE20K, Cityscapes and COCO-stuff, which are important benchmarks for semantic segmentation. All checkpoints can be\n  found on the [hub](https://huggingface.co/models?other=segformer).\n- The quickest way to get started with SegFormer is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer) (which showcase both inference and\n  fine-tuning on custom data). One can also check out the [blog post](https://huggingface.co/blog/fine-tune-segformer) introducing SegFormer and illustrating how it can be fine-tuned on custom data.\n- TensorFlow users should refer to [this repository](https://github.com/deep-diver/segformer-tf-transformers) that shows off-the-shelf inference and fine-tuning.\n- One can also check out [this interactive demo on Hugging Face Spaces](https://huggingface.co/spaces/chansung/segformer-tf-transformers)\n  to try out a SegFormer model on custom images.\n- SegFormer works on any input size, as it pads the input to be divisible by `config.patch_sizes`.\n- One can use [`SegformerImageProcessor`] to prepare images and corresponding segmentation maps\n  for the model. Note that this image processor is fairly basic and does not include all data augmentations used in\n  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most\n  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,\n  such as 512x512 or 640x640, after which they are normalized.\n- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with\n  `do_reduce_labels` set to `True` or `False`."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4a9289ad-e512-4d53-ba8a-d3bd00616d57
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer/", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4a9289ad-e512-4d53-ba8a-d3bd00616d57\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"faa84358-f68c-49f6-9c18-aa296da38ff4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b3c987491b30eab9215da2400488288fc697783bb4c83b9d84d691b8ce54f26\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c34ca2f4-894f-4f01-9a2e-2a421521b949\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"266f7af805217401a72e7b0c8cde47c959fb8729895220a0fbc5f48838cfcc08\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db527d9c-ff12-4006-b91c-08d4c5c4e16f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46576fe9aea188e6b532d6b6f1a1dae1a076ba0c16b7649ffbaa395e43fef3c8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3178, \"end_char_idx\": 5659, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "ref_doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "chroma:document": "- The quickest way to get started with SegFormer is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer) (which showcase both inference and\n  fine-tuning on custom data). One can also check out the [blog post](https://huggingface.co/blog/fine-tune-segformer) introducing SegFormer and illustrating how it can be fine-tuned on custom data.\n- TensorFlow users should refer to [this repository](https://github.com/deep-diver/segformer-tf-transformers) that shows off-the-shelf inference and fine-tuning.\n- One can also check out [this interactive demo on Hugging Face Spaces](https://huggingface.co/spaces/chansung/segformer-tf-transformers)\n  to try out a SegFormer model on custom images.\n- SegFormer works on any input size, as it pads the input to be divisible by `config.patch_sizes`.\n- One can use [`SegformerImageProcessor`] to prepare images and corresponding segmentation maps\n  for the model. Note that this image processor is fairly basic and does not include all data augmentations used in\n  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most\n  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,\n  such as 512x512 or 640x640, after which they are normalized.\n- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with\n  `do_reduce_labels` set to `True` or `False`. In some datasets (like ADE20k), the 0 index is used in the annotated\n  segmentation maps for background. However, ADE20k doesn't include the \"background\" class in its 150 labels.\n  Therefore, `do_reduce_labels` is used to reduce all labels by 1, and to make sure no loss is computed for the\n  background class (i.e. it replaces 0 in the annotated maps by 255, which is the *ignore_index* of the loss function\n  used by [`SegformerForSemanticSegmentation`]). However, other datasets use the 0 index as\n  background class and include this class as part of all labels. In that case, `do_reduce_labels` should be set to\n  `False`, as loss should also be computed for the background class.\n- As most models, SegFormer comes in different sizes, the details of which can be found in the table below\n  (taken from Table 7 of the [original paper](https://arxiv.org/abs/2105.15203))."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: db527d9c-ff12-4006-b91c-08d4c5c4e16f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer/", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"db527d9c-ff12-4006-b91c-08d4c5c4e16f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"faa84358-f68c-49f6-9c18-aa296da38ff4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b3c987491b30eab9215da2400488288fc697783bb4c83b9d84d691b8ce54f26\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4a9289ad-e512-4d53-ba8a-d3bd00616d57\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"166e5468f14a50f22edb23bb267dd49392a93ae9ff53580e66dfae60ea17df63\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fa4fa4d-fca9-4ffe-a066-1652583c5ee4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f30e53434db5327c93966eeb77321e6568c73e00f85384acb87057ed9d0547c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4137, \"end_char_idx\": 6782, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "ref_doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "chroma:document": "Note that this image processor is fairly basic and does not include all data augmentations used in\n  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most\n  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,\n  such as 512x512 or 640x640, after which they are normalized.\n- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with\n  `do_reduce_labels` set to `True` or `False`. In some datasets (like ADE20k), the 0 index is used in the annotated\n  segmentation maps for background. However, ADE20k doesn't include the \"background\" class in its 150 labels.\n  Therefore, `do_reduce_labels` is used to reduce all labels by 1, and to make sure no loss is computed for the\n  background class (i.e. it replaces 0 in the annotated maps by 255, which is the *ignore_index* of the loss function\n  used by [`SegformerForSemanticSegmentation`]). However, other datasets use the 0 index as\n  background class and include this class as part of all labels. In that case, `do_reduce_labels` should be set to\n  `False`, as loss should also be computed for the background class.\n- As most models, SegFormer comes in different sizes, the details of which can be found in the table below\n  (taken from Table 7 of the [original paper](https://arxiv.org/abs/2105.15203)).\n\n| **Model variant** | **Depths**    | **Hidden sizes**    | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :---------------------: | :------------: | :-------------------: |\n| MiT-b0            | [2, 2, 2, 2]  | [32, 64, 160, 256]  | 256                     | 3.7            | 70.5                  |\n| MiT-b1            | [2, 2, 2, 2]  | [64, 128, 320, 512] | 256                     | 14.0           | 78.7                  |\n| MiT-b2            | [3, 4, 6, 3]  | [64, 128, 320, 512] | 768                     | 25.4           | 81.6                  |\n| MiT-b3            | [3, 4, 18, 3] | [64, 128, 320, 512] | 768                     | 45.2           | 83.1                  |\n| MiT-b4            | [3, 8, 27, 3] | [64, 128, 320, 512] | 768                     | 62.6           | 83.6                  |\n| MiT-b5            | [3, 6, 40, 3] | [64, 128, 320, 512] | 768                     | 82.0           | 83.8                  |\n\nNote that MiT in the above table refers to the Mix Transformer encoder backbone introduced in SegFormer."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7fa4fa4d-fca9-4ffe-a066-1652583c5ee4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer/", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7fa4fa4d-fca9-4ffe-a066-1652583c5ee4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"faa84358-f68c-49f6-9c18-aa296da38ff4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b3c987491b30eab9215da2400488288fc697783bb4c83b9d84d691b8ce54f26\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db527d9c-ff12-4006-b91c-08d4c5c4e16f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"57b6bcbad7564b053f948d3f985abbc8672185263068a60fc40ae7e7662a3d36\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d357df47-fedd-4a4f-a6aa-f2f8376282ec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc0f3945eafb535d4a7deb531a0249224ee1918e045a2b964b3483e6a5c1beb8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5661, \"end_char_idx\": 8418, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "ref_doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "chroma:document": "| **Model variant** | **Depths**    | **Hidden sizes**    | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :---------------------: | :------------: | :-------------------: |\n| MiT-b0            | [2, 2, 2, 2]  | [32, 64, 160, 256]  | 256                     | 3.7            | 70.5                  |\n| MiT-b1            | [2, 2, 2, 2]  | [64, 128, 320, 512] | 256                     | 14.0           | 78.7                  |\n| MiT-b2            | [3, 4, 6, 3]  | [64, 128, 320, 512] | 768                     | 25.4           | 81.6                  |\n| MiT-b3            | [3, 4, 18, 3] | [64, 128, 320, 512] | 768                     | 45.2           | 83.1                  |\n| MiT-b4            | [3, 8, 27, 3] | [64, 128, 320, 512] | 768                     | 62.6           | 83.6                  |\n| MiT-b5            | [3, 6, 40, 3] | [64, 128, 320, 512] | 768                     | 82.0           | 83.8                  |\n\nNote that MiT in the above table refers to the Mix Transformer encoder backbone introduced in SegFormer. For\nSegFormer's results on the segmentation datasets like ADE20k, refer to the [paper](https://arxiv.org/abs/2105.15203).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SegFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SegformerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- [Image classification task guide](../tasks/image_classification)\n\nSemantic segmentation:\n\n- [`SegformerForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation).\n- A blog on fine-tuning SegFormer on a custom dataset can be found [here](https://huggingface.co/blog/fine-tune-segformer).\n- More demo notebooks on SegFormer (both inference + fine-tuning on a custom dataset) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).\n- [`TFSegformerForSemanticSegmentation`] is supported by this [example notebook](https://github.com/huggingface/notebooks/blob/main/examples/semantic_segmentation-tf.ipynb).\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d357df47-fedd-4a4f-a6aa-f2f8376282ec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer/", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d357df47-fedd-4a4f-a6aa-f2f8376282ec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"faa84358-f68c-49f6-9c18-aa296da38ff4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2b3c987491b30eab9215da2400488288fc697783bb4c83b9d84d691b8ce54f26\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7fa4fa4d-fca9-4ffe-a066-1652583c5ee4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer/\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9546d80d30e4bde7b3419deb23fd6845442e51d27bf1564719dd8b5d1523f9c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6783, \"end_char_idx\": 9431, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "ref_doc_id": "faa84358-f68c-49f6-9c18-aa296da38ff4", "chroma:document": "For\nSegFormer's results on the segmentation datasets like ADE20k, refer to the [paper](https://arxiv.org/abs/2105.15203).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SegFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SegformerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- [Image classification task guide](../tasks/image_classification)\n\nSemantic segmentation:\n\n- [`SegformerForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation).\n- A blog on fine-tuning SegFormer on a custom dataset can be found [here](https://huggingface.co/blog/fine-tune-segformer).\n- More demo notebooks on SegFormer (both inference + fine-tuning on a custom dataset) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).\n- [`TFSegformerForSemanticSegmentation`] is supported by this [example notebook](https://github.com/huggingface/notebooks/blob/main/examples/semantic_segmentation-tf.ipynb).\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## SegformerConfig\n\n[[autodoc]] SegformerConfig\n\n## SegformerFeatureExtractor\n\n[[autodoc]] SegformerFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## SegformerImageProcessor\n\n[[autodoc]] SegformerImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n<frameworkcontent>\n<pt>\n\n## SegformerModel\n\n[[autodoc]] SegformerModel\n    - forward\n\n## SegformerDecodeHead\n\n[[autodoc]] SegformerDecodeHead\n    - forward\n\n## SegformerForImageClassification\n\n[[autodoc]] SegformerForImageClassification\n    - forward\n\n## SegformerForSemanticSegmentation\n\n[[autodoc]] SegformerForSemanticSegmentation\n    - forward\n\n</pt>\n<tf>\n\n## TFSegformerDecodeHead\n\n[[autodoc]] TFSegformerDecodeHead\n    - call\n\n## TFSegformerModel\n\n[[autodoc]] TFSegformerModel\n    - call\n\n## TFSegformerForImageClassification\n\n[[autodoc]] TFSegformerForImageClassification\n    - call\n\n## TFSegformerForSemanticSegmentation\n\n[[autodoc]] TFSegformerForSemanticSegmentation\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 64494697-a469-4d3f-bd5c-3741568599db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xls_r/", "title": "XLS-R", "tokens": 705, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"64494697-a469-4d3f-bd5c-3741568599db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xls_r/\", \"title\": \"XLS-R\", \"tokens\": 705, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5258b7f1-3b77-48fc-8812-310429943477\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xls_r/\", \"title\": \"XLS-R\", \"tokens\": 705, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"935de1077f017ed09ade9fe4c3460ee70321ca6644f711c9e620a19f60937e08\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2170, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5258b7f1-3b77-48fc-8812-310429943477", "doc_id": "5258b7f1-3b77-48fc-8812-310429943477", "ref_doc_id": "5258b7f1-3b77-48fc-8812-310429943477", "chroma:document": "# XLS-R\n\n## Overview\n\nThe XLS-R model was proposed in [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://arxiv.org/abs/2111.09296) by Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman\nGoyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli.\n\nThe abstract from the paper is the following:\n\n*This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0.\nWe train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128\nlanguages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range\nof tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation\nbenchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into\nEnglish. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as\nVoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107\nlanguage identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform\nEnglish-only pretraining when translating English speech into other languages, a setting which favors monolingual\npretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.*\n\nRelevant checkpoints can be found under https://huggingface.co/models?other=xls_r.\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).\n\n## Usage tips\n\n- XLS-R is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- XLS-R model was trained using connectionist temporal classification (CTC) so the model output has to be decoded using\n  [`Wav2Vec2CTCTokenizer`].\n\n<Tip>\n\nXLS-R's architecture is based on the Wav2Vec2 model, refer to [Wav2Vec2's documentation page](wav2vec2) for API reference.\n\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ac9d53de-636e-45d7-a065-97a4c7e16eef
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-v/", "title": "XLM-V", "tokens": 706, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ac9d53de-636e-45d7-a065-97a4c7e16eef\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-v/\", \"title\": \"XLM-V\", \"tokens\": 706, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b63818cb-f2ce-4f6d-9205-47631c58b3d9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-v/\", \"title\": \"XLM-V\", \"tokens\": 706, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d977194c6e14c27f0d1ed27638bbdf8a18a751a001496a7b26ce9ad8e668ef6a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b63818cb-f2ce-4f6d-9205-47631c58b3d9", "doc_id": "b63818cb-f2ce-4f6d-9205-47631c58b3d9", "ref_doc_id": "b63818cb-f2ce-4f6d-9205-47631c58b3d9", "chroma:document": "# XLM-V\n\n## Overview\n\nXLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R).\nIt was introduced in the [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://arxiv.org/abs/2301.10472)\npaper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa.\n\nFrom the abstract of the XLM-V paper:\n\n*Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.\nAs these models have increased in parameter count and depth, vocabulary size has remained largely unchanged.\nThis vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R.\nIn this paper, we introduce a new approach for scaling to very large multilingual vocabularies by\nde-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity\nto achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically\nmore semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V,\na multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we\ntested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and\nnamed entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).*\n\nThis model was contributed by [stefan-it](https://huggingface.co/stefan-it), including detailed experiments with XLM-V on downstream tasks.\nThe experiments repository can be found [here](https://github.com/stefan-it/xlm-v-experiments).\n\n## Usage tips\n\n- XLM-V is compatible with the XLM-RoBERTa model architecture, only model weights from [`fairseq`](https://github.com/facebookresearch/fairseq)\n  library had to be converted.\n- The `XLMTokenizer` implementation is used to load the vocab and performs tokenization.\n\nA XLM-V (base size) model is available under the [`facebook/xlm-v-base`](https://huggingface.co/facebook/xlm-v-base) identifier.\n\n<Tip>\n\nXLM-V architecture is the same as XLM-RoBERTa, refer to [XLM-RoBERTa documentation](xlm-roberta) for API reference, and examples.\n</Tip>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4dc3446c-28a8-4a84-a1c9-96739db798be
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rembert/", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4dc3446c-28a8-4a84-a1c9-96739db798be\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert/\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"53e69a29-3a83-419c-b33d-b33f3a5b82e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert/\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fced0cf008ea20507128fd73336992e52e39feb24d53f1d5309c400d60e0896c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"39514926-6b01-4b7e-b65c-a528dc0fbfbf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c8e97a119d7ccfd449e3f9e46958d008acf9378dc4ebd41c8edab5cf1b3063ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1894, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "53e69a29-3a83-419c-b33d-b33f3a5b82e3", "doc_id": "53e69a29-3a83-419c-b33d-b33f3a5b82e3", "ref_doc_id": "53e69a29-3a83-419c-b33d-b33f3a5b82e3", "chroma:document": "# RemBERT\n\n## Overview\n\nThe RemBERT model was proposed in [Rethinking Embedding Coupling in Pre-trained Language Models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault F\u00e9vry, Henry Tsai, Melvin Johnson, Sebastian Ruder.\n\nThe abstract from the paper is the following:\n\n*We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art\npre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to\nsignificantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By\nreallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on\nstandard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that\nallocating additional capacity to the output embedding provides benefits to the model that persist through the\nfine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger\noutput embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage\nTransformer representations to be more general and more transferable to other tasks and languages. Harnessing these\nfindings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the\nnumber of parameters at the fine-tuning stage.*\n\n## Usage tips\n\nFor fine-tuning, RemBERT can be thought of as a bigger version of mBERT with an ALBERT-like factorization of the\nembedding layer. The embeddings are not tied in pre-training, in contrast with BERT, which enables smaller input\nembeddings (preserved during fine-tuning) and bigger output embeddings (discarded at fine-tuning). The tokenizer is\nalso similar to the Albert one rather than the BERT one."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 39514926-6b01-4b7e-b65c-a528dc0fbfbf
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rembert/", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"39514926-6b01-4b7e-b65c-a528dc0fbfbf\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert/\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"53e69a29-3a83-419c-b33d-b33f3a5b82e3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert/\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"fced0cf008ea20507128fd73336992e52e39feb24d53f1d5309c400d60e0896c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4dc3446c-28a8-4a84-a1c9-96739db798be\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert/\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0ffb7b37de5833d9bf0f9ee899a1033121bff0b5640f4735cc33d2126d462948\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3904, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "53e69a29-3a83-419c-b33d-b33f3a5b82e3", "doc_id": "53e69a29-3a83-419c-b33d-b33f3a5b82e3", "ref_doc_id": "53e69a29-3a83-419c-b33d-b33f3a5b82e3", "chroma:document": "# RemBERT\n\n## Overview\n\nThe RemBERT model was proposed in [Rethinking Embedding Coupling in Pre-trained Language Models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault F\u00e9vry, Henry Tsai, Melvin Johnson, Sebastian Ruder.\n\nThe abstract from the paper is the following:\n\n*We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art\npre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to\nsignificantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By\nreallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on\nstandard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that\nallocating additional capacity to the output embedding provides benefits to the model that persist through the\nfine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger\noutput embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage\nTransformer representations to be more general and more transferable to other tasks and languages. Harnessing these\nfindings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the\nnumber of parameters at the fine-tuning stage.*\n\n## Usage tips\n\nFor fine-tuning, RemBERT can be thought of as a bigger version of mBERT with an ALBERT-like factorization of the\nembedding layer. The embeddings are not tied in pre-training, in contrast with BERT, which enables smaller input\nembeddings (preserved during fine-tuning) and bigger output embeddings (discarded at fine-tuning). The tokenizer is\nalso similar to the Albert one rather than the BERT one.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RemBertConfig\n\n[[autodoc]] RemBertConfig\n\n## RemBertTokenizer\n\n[[autodoc]] RemBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RemBertTokenizerFast\n\n[[autodoc]] RemBertTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## RemBertModel\n\n[[autodoc]] RemBertModel\n    - forward\n\n## RemBertForCausalLM\n\n[[autodoc]] RemBertForCausalLM\n    - forward\n\n## RemBertForMaskedLM\n\n[[autodoc]] RemBertForMaskedLM\n    - forward\n\n## RemBertForSequenceClassification\n\n[[autodoc]] RemBertForSequenceClassification\n    - forward\n\n## RemBertForMultipleChoice\n\n[[autodoc]] RemBertForMultipleChoice\n    - forward\n\n## RemBertForTokenClassification\n\n[[autodoc]] RemBertForTokenClassification\n    - forward\n\n## RemBertForQuestionAnswering\n\n[[autodoc]] RemBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRemBertModel\n\n[[autodoc]] TFRemBertModel\n    - call\n\n## TFRemBertForMaskedLM\n\n[[autodoc]] TFRemBertForMaskedLM\n    - call\n\n## TFRemBertForCausalLM\n\n[[autodoc]] TFRemBertForCausalLM\n    - call\n\n## TFRemBertForSequenceClassification\n\n[[autodoc]] TFRemBertForSequenceClassification\n    - call\n\n## TFRemBertForMultipleChoice\n\n[[autodoc]] TFRemBertForMultipleChoice\n    - call\n\n## TFRemBertForTokenClassification\n\n[[autodoc]] TFRemBertForTokenClassification\n    - call\n\n## TFRemBertForQuestionAnswering\n\n[[autodoc]] TFRemBertForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b1cbbfd1-5346-4764-8cf5-3fc306d2731a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_moe/", "title": "Qwen2MoE", "tokens": 941, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b1cbbfd1-5346-4764-8cf5-3fc306d2731a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_moe/\", \"title\": \"Qwen2MoE\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4d30f231-4258-4a85-b57f-82d8b3d0ef83\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_moe/\", \"title\": \"Qwen2MoE\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5e894d2a121de8b864c705855148f8bb94cad946292ffe830592c55d85fd64ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2797, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4d30f231-4258-4a85-b57f-82d8b3d0ef83", "doc_id": "4d30f231-4258-4a85-b57f-82d8b3d0ef83", "ref_doc_id": "4d30f231-4258-4a85-b57f-82d8b3d0ef83", "chroma:document": "# Qwen2MoE\n\n## Overview\n\nQwen2MoE is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen-72B, Qwen-1.8B, Qwen-VL, Qwen-Audio, etc.\n\n### Model Details\n\nQwen2MoE is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. Qwen2MoE has the following architectural choices:\n\n- Qwen2MoE is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.\n- Qwen2MoE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, `Qwen1.5-MoE-A2.7B` is upcycled from `Qwen-1.8B`. It has 14.3B parameters in total and 2.7B activated parameters during runtime, while it achieves comparable performance with `Qwen1.5-7B`, with only 25% of the training resources.\n\nFor more details refer to the [release blog post](https://qwenlm.github.io/blog/qwen-moe/).\n\n## Usage tips\n\n`Qwen1.5-MoE-A2.7B` and `Qwen1.5-MoE-A2.7B-Chat` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)\n\nIn the following, we demonstrate how to use `Qwen1.5-MoE-A2.7B-Chat` for the inference. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen1.5-MoE-A2.7B-Chat\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen1.5-MoE-A2.7B-Chat\")\n\n>>> prompt = \"Give me a short introduction to large language model.\"\n\n>>> messages = [{\"role\": \"user\", \"content\": prompt}]\n\n>>> text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n\n>>> model_inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n\n>>> generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)\n\n>>> generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]\n\n>>> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\n## Qwen2MoeConfig\n\n[[autodoc]] Qwen2MoeConfig\n\n## Qwen2MoeModel\n\n[[autodoc]] Qwen2MoeModel\n    - forward\n\n## Qwen2MoeForCausalLM\n\n[[autodoc]] Qwen2MoeForCausalLM\n    - forward\n\n## Qwen2MoeForSequenceClassification\n\n[[autodoc]] Qwen2MoeForSequenceClassification\n    - forward\n\n## Qwen2MoeForTokenClassification\n\n[[autodoc]] Qwen2MoeForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: af756720-4323-4126-bea2-6c80061c6579
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"af756720-4323-4126-bea2-6c80061c6579\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b2547e82-2625-4fb2-a3b1-7da94c91a2da\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c86392f-9e32-458a-8de1-cbef0cbdb197\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c545c881b354c5d2123665c77a39fa2246c915604548056bcb996a838e31c2ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3388, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "ref_doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "chroma:document": "# SeamlessM4T\n\n## Overview\n\nThe SeamlessM4T model was proposed in [SeamlessM4T \u2014 Massively Multilingual & Multimodal Machine Translation](https://dl.fbaipublicfiles.com/seamless/seamless_m4t_paper.pdf) by the Seamless Communication team from Meta AI.\n\nThis is the **version 1** release of the model. For the updated **version 2** release, refer to the [Seamless M4T v2 docs](https://huggingface.co/docs/transformers/main/model_doc/seamless_m4t_v2).\n\nSeamlessM4T is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text.\n\nSeamlessM4T enables multiple tasks without relying on separate models:\n\n- Speech-to-speech translation (S2ST)\n- Speech-to-text translation (S2TT)\n- Text-to-speech translation (T2ST)\n- Text-to-text translation (T2TT)\n- Automatic speech recognition (ASR)\n\n[`SeamlessM4TModel`] can perform all the above tasks, but each task also has its own dedicated sub-model.\n\nThe abstract from the paper is the following:\n\n*What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication*\n\n## Usage\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4TModel\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n>>> model = SeamlessM4TModel.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3c86392f-9e32-458a-8de1-cbef0cbdb197
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3c86392f-9e32-458a-8de1-cbef0cbdb197\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b2547e82-2625-4fb2-a3b1-7da94c91a2da\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"af756720-4323-4126-bea2-6c80061c6579\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"0bac659db3c4d61cfe6695db433519c51d223b512bb51bdfd25eb64cc600460b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1fabf5e2-7347-40a2-b484-9c741f1526d4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f3fd281d257beb3f8ad6c171d794c992afe58c24c2b5508bb87f8d03d8c5c086\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1800, \"end_char_idx\": 5077, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "ref_doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "chroma:document": "To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication*\n\n## Usage\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4TModel\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n>>> model = SeamlessM4TModel.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio.\n\nHere is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English test as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```\n\n\n### Speech\n\n[`SeamlessM4TModel`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4TModel.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1fabf5e2-7347-40a2-b484-9c741f1526d4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1fabf5e2-7347-40a2-b484-9c741f1526d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b2547e82-2625-4fb2-a3b1-7da94c91a2da\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c86392f-9e32-458a-8de1-cbef0cbdb197\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"be33a222ed163f22101bd4351909cb4572fc1e2cfc6a616417073213e00210e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"52d9d5e3-8628-4ec8-a0d3-9c69543c5cd5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0baafe24a31e2b0559b9a0369a6c30203f6eef4fb92f3b5d0e655aed494f848b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3390, \"end_char_idx\": 6745, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "ref_doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "chroma:document": "Here is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English test as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```\n\n\n### Speech\n\n[`SeamlessM4TModel`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4TModel.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips\n\n\n#### 1. Use dedicated models\n\n[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.\nFor example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: \n\n```python\n>>> from transformers import SeamlessM4TForSpeechToSpeech\n>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nOr you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.\n\n```python\n>>> from transformers import SeamlessM4TForTextToText\n>>> model = SeamlessM4TForTextToText.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nFeel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !\n\n## Model architecture"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 52d9d5e3-8628-4ec8-a0d3-9c69543c5cd5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"52d9d5e3-8628-4ec8-a0d3-9c69543c5cd5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b2547e82-2625-4fb2-a3b1-7da94c91a2da\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1fabf5e2-7347-40a2-b484-9c741f1526d4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a06facd3acf8d87f27a84db2737bb0ea9532b4d8ef61118efa840484cb54362\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"21916a5e-14be-491b-aa25-b738a9d45ae7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"31224a1abbd7ba94ccb28d25bb81b1aeacfa787a9a29eafa2a9f43532e08eae4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5080, \"end_char_idx\": 8138, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "ref_doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "chroma:document": "#### 1. Use dedicated models\n\n[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.\nFor example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: \n\n```python\n>>> from transformers import SeamlessM4TForSpeechToSpeech\n>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nOr you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.\n\n```python\n>>> from transformers import SeamlessM4TForTextToText\n>>> model = SeamlessM4TForTextToText.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nFeel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !\n\n## Model architecture\n\n\nSeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text.\n\nEach modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.\n- These unit tokens are then passed through the final vocoder to produce the actual speech.\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## SeamlessM4TModel\n\n[[autodoc]] SeamlessM4TModel\n    - generate\n\n\n## SeamlessM4TForTextToSpeech\n\n[[autodoc]] SeamlessM4TForTextToSpeech\n    - generate\n\n\n## SeamlessM4TForSpeechToSpeech\n\n[[autodoc]] SeamlessM4TForSpeechToSpeech\n    - generate"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 21916a5e-14be-491b-aa25-b738a9d45ae7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t/", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"21916a5e-14be-491b-aa25-b738a9d45ae7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b2547e82-2625-4fb2-a3b1-7da94c91a2da\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6c4b1f997a6962b1d1537453982eb9d1d569755f4cd1404b16170229268162d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"52d9d5e3-8628-4ec8-a0d3-9c69543c5cd5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t/\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6efe8c5cd5c617582c0df358c204650a01f4860ec6edd89161ffe88e3c5039d9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6748, \"end_char_idx\": 9167, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "ref_doc_id": "b2547e82-2625-4fb2-a3b1-7da94c91a2da", "chroma:document": "SeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text.\n\nEach modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.\n- These unit tokens are then passed through the final vocoder to produce the actual speech.\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## SeamlessM4TModel\n\n[[autodoc]] SeamlessM4TModel\n    - generate\n\n\n## SeamlessM4TForTextToSpeech\n\n[[autodoc]] SeamlessM4TForTextToSpeech\n    - generate\n\n\n## SeamlessM4TForSpeechToSpeech\n\n[[autodoc]] SeamlessM4TForSpeechToSpeech\n    - generate\n\n\n## SeamlessM4TForTextToText\n\n[[autodoc]] transformers.SeamlessM4TForTextToText\n    - forward\n    - generate\n\n## SeamlessM4TForSpeechToText\n\n[[autodoc]] transformers.SeamlessM4TForSpeechToText\n    - forward\n    - generate\n\n## SeamlessM4TConfig\n\n[[autodoc]] SeamlessM4TConfig\n\n\n## SeamlessM4TTokenizer\n\n[[autodoc]] SeamlessM4TTokenizer\n    - __call__\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n\n## SeamlessM4TTokenizerFast\n\n[[autodoc]] SeamlessM4TTokenizerFast\n    - __call__\n\n## SeamlessM4TFeatureExtractor\n\n[[autodoc]] SeamlessM4TFeatureExtractor\n    - __call__\n\n## SeamlessM4TProcessor\n\n[[autodoc]] SeamlessM4TProcessor\n    - __call__\n\n## SeamlessM4TCodeHifiGan\n\n[[autodoc]] SeamlessM4TCodeHifiGan\n\n\n## SeamlessM4THifiGan\n\n[[autodoc]] SeamlessM4THifiGan\n\n## SeamlessM4TTextToUnitModel\n\n[[autodoc]] SeamlessM4TTextToUnitModel\n\n## SeamlessM4TTextToUnitForConditionalGeneration\n\n[[autodoc]] SeamlessM4TTextToUnitForConditionalGeneration"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3c1b76bf-1540-4273-868a-8f37cc72386f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3c1b76bf-1540-4273-868a-8f37cc72386f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"793fb8c1-c8f3-4154-813f-223b697b66ce\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adb2c51d-8478-4bc1-974c-e44d5b94e168\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e20dafc9d6623ee78a309cc42aabc93d29c085b753dc87c9290905ac1c24940e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "ref_doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "chroma:document": "# ImageGPT\n\n## Overview\n\nThe ImageGPT model was proposed in [Generative Pretraining from Pixels](https://openai.com/blog/image-gpt) by Mark\nChen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever. ImageGPT (iGPT) is a GPT-2-like\nmodel trained to predict the next pixel value, allowing for both unconditional and conditional image generation.\n\nThe abstract from the paper is the following:\n\n*Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models\ncan learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels,\nwithout incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels,\nwe find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and\nlow-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide\nResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pre-trained models. We are also\ncompetitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0%\ntop-1 accuracy on a linear probe of our features.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/imagegpt_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Summary of the approach. Taken from the [original paper](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf). </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr), based on [this issue](https://github.com/openai/image-gpt/issues/7). The original code can be found\n[here](https://github.com/openai/image-gpt).\n\n## Usage tips\n\n- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation\n  function is used (namely \"quick gelu\"), and the layer normalization layers don't mean center the inputs. ImageGPT\n  also doesn't have tied input- and output embeddings.\n- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence\n  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a\n  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors\n  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long\n  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger\n  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special \"start of sentence\" (SOS)\n  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare\n  images for the model.\n- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly\n  performant image features useful for downstream tasks, such as image classification."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: adb2c51d-8478-4bc1-974c-e44d5b94e168
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"adb2c51d-8478-4bc1-974c-e44d5b94e168\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"793fb8c1-c8f3-4154-813f-223b697b66ce\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c1b76bf-1540-4273-868a-8f37cc72386f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1c903dbded76142f66fa13735b7f8315af44ff61c89e65ef3baab87084f13440\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"529294bd-5da9-4952-81b9-e76a7dbc5f9b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5fccfa84855df62b8f1903509f9f52776a67d488581354f7e837cf743acf608c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1717, \"end_char_idx\": 3883, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "ref_doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "chroma:document": "The original code can be found\n[here](https://github.com/openai/image-gpt).\n\n## Usage tips\n\n- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation\n  function is used (namely \"quick gelu\"), and the layer normalization layers don't mean center the inputs. ImageGPT\n  also doesn't have tied input- and output embeddings.\n- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence\n  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a\n  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors\n  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long\n  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger\n  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special \"start of sentence\" (SOS)\n  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare\n  images for the model.\n- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly\n  performant image features useful for downstream tasks, such as image classification. The authors showed that the\n  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as\n  a sklearn logistic regression model for example). This is also referred to as \"linear probing\". Features can be\n  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and\n  then average-pool the hidden states at whatever layer you like.\n- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can\n  use [`ImageGPTForImageClassification`].\n- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also\n  train an XL variant, which they didn't release."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 529294bd-5da9-4952-81b9-e76a7dbc5f9b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"529294bd-5da9-4952-81b9-e76a7dbc5f9b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"793fb8c1-c8f3-4154-813f-223b697b66ce\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adb2c51d-8478-4bc1-974c-e44d5b94e168\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6cbbee26956adc8a0e8b34c0882607d4ed370b04c41c28f5e86a846bbb23f6b2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cc4da21d-0b98-405d-9982-dfa6b5425bb8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"42c5f4faa6c1ce043a0454ff6de7536d5d55d600c806f46e56d7872e218dedbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2300, \"end_char_idx\": 4630, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "ref_doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "chroma:document": "However, feeding a\n  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors\n  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long\n  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger\n  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special \"start of sentence\" (SOS)\n  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare\n  images for the model.\n- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly\n  performant image features useful for downstream tasks, such as image classification. The authors showed that the\n  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as\n  a sklearn logistic regression model for example). This is also referred to as \"linear probing\". Features can be\n  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and\n  then average-pool the hidden states at whatever layer you like.\n- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can\n  use [`ImageGPTForImageClassification`].\n- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also\n  train an XL variant, which they didn't release. The differences in size are summarized in the following table:\n\n| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n|---|---|---|---|---|---|\n| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |\n| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |\n| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |\n| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |\n| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |\n| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ImageGPT."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cc4da21d-0b98-405d-9982-dfa6b5425bb8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt/", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cc4da21d-0b98-405d-9982-dfa6b5425bb8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"793fb8c1-c8f3-4154-813f-223b697b66ce\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8c142cc3f75258afc813c339fa6b17e5c4bf7e272a9df2ae07ae29cc081cfc45\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"529294bd-5da9-4952-81b9-e76a7dbc5f9b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt/\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"db95983905acb49a976ff2b4ed598080f1973a87f23852e3790e8dac13d436a2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3813, \"end_char_idx\": 5858, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "ref_doc_id": "793fb8c1-c8f3-4154-813f-223b697b66ce", "chroma:document": "The authors did also\n  train an XL variant, which they didn't release. The differences in size are summarized in the following table:\n\n| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n|---|---|---|---|---|---|\n| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |\n| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |\n| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |\n| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |\n| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |\n| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ImageGPT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- Demo notebooks for ImageGPT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ImageGPT).\n- [`ImageGPTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ImageGPTConfig\n\n[[autodoc]] ImageGPTConfig\n\n## ImageGPTFeatureExtractor\n\n[[autodoc]] ImageGPTFeatureExtractor\n    - __call__\n\n## ImageGPTImageProcessor\n\n[[autodoc]] ImageGPTImageProcessor\n    - preprocess\n\n## ImageGPTModel\n\n[[autodoc]] ImageGPTModel\n    - forward\n\n## ImageGPTForCausalImageModeling\n\n[[autodoc]] ImageGPTForCausalImageModeling\n    - forward\n\n## ImageGPTForImageClassification\n\n[[autodoc]] ImageGPTForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d5ea83f8-899a-4cf7-ab9a-70480f467a63
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nezha/", "title": "Nezha", "tokens": 857, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d5ea83f8-899a-4cf7-ab9a-70480f467a63\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nezha/\", \"title\": \"Nezha\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7fca0363-491d-4a12-982b-f88fee9d1dc7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nezha/\", \"title\": \"Nezha\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9133059a4c28828a08cbe8358c36f2e56f7596e1810d73b28d9b51f2d8a0a3bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2821, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7fca0363-491d-4a12-982b-f88fee9d1dc7", "doc_id": "7fca0363-491d-4a12-982b-f88fee9d1dc7", "ref_doc_id": "7fca0363-491d-4a12-982b-f88fee9d1dc7", "chroma:document": "# Nezha\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe Nezha model was proposed in [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://arxiv.org/abs/1909.00204) by Junqiu Wei et al.\n\nThe abstract from the paper is the following:\n\n*The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks\ndue to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.\nIn this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed\nrepresentation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks.\nThe current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional\nRelative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy,\nMixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA\nachieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including\nnamed entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti)\nand natural language inference (XNLI).*\n\nThis model was contributed by [sijunhe](https://huggingface.co/sijunhe). The original code can be found [here](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-PyTorch).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## NezhaConfig\n\n[[autodoc]] NezhaConfig\n\n## NezhaModel\n\n[[autodoc]] NezhaModel\n    - forward\n\n## NezhaForPreTraining\n\n[[autodoc]] NezhaForPreTraining\n    - forward\n\n## NezhaForMaskedLM\n\n[[autodoc]] NezhaForMaskedLM\n    - forward\n\n## NezhaForNextSentencePrediction\n\n[[autodoc]] NezhaForNextSentencePrediction\n    - forward\n\n## NezhaForSequenceClassification\n\n[[autodoc]] NezhaForSequenceClassification\n    - forward\n\n## NezhaForMultipleChoice\n\n[[autodoc]] NezhaForMultipleChoice\n    - forward\n\n## NezhaForTokenClassification\n\n[[autodoc]] NezhaForTokenClassification\n    - forward\n\n## NezhaForQuestionAnswering\n\n[[autodoc]] NezhaForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c1885d98-77db-40e1-9cf1-02f320e05b8a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c1885d98-77db-40e1-9cf1-02f320e05b8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8fc5b214-4f01-43d3-8474-dc1d71078291\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c60a644fba1fa1a6d31884cc274859929434cc364eb0fa31f56b8f8ed0726a90\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40ff3b4e-6223-4539-bc91-aaf28c4d01a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4e4b6db29d7225bf740abda8c406d28516077dde573d58a7868566c3b43b491\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "doc_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "ref_doc_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "chroma:document": "# Audio Spectrogram Transformer\n\n## Overview\n\nThe Audio Spectrogram Transformer model was proposed in [AST: Audio Spectrogram Transformer](https://arxiv.org/abs/2104.01778) by Yuan Gong, Yu-An Chung, James Glass.\nThe Audio Spectrogram Transformer applies a [Vision Transformer](vit) to audio, by turning audio into an image (spectrogram). The model obtains state-of-the-art results\nfor audio classification.\n\nThe abstract from the paper is the following:\n\n*In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Audio Spectrogram Transformer architecture. Taken from the <a href=\"https://arxiv.org/abs/2104.01778\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/YuanGongND/ast).\n\n## Usage tips\n\n- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make\nsure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet\nmean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how\nthe authors compute the stats for a downstream dataset.\n- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the\n[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 40ff3b4e-6223-4539-bc91-aaf28c4d01a5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"40ff3b4e-6223-4539-bc91-aaf28c4d01a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8fc5b214-4f01-43d3-8474-dc1d71078291\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c60a644fba1fa1a6d31884cc274859929434cc364eb0fa31f56b8f8ed0726a90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c1885d98-77db-40e1-9cf1-02f320e05b8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8a182a06b3f56a8fcdb9edef652a01ffa28a98f2681e7b284cbd46f3436eff9f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec6dcd84-36d9-4f9c-bc18-11eb71d6baf2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d901af1d1cdc7253b73f04dc6ef84ddb1b30a0d78d7c3f3bc7d5c81caeba8116\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1651, \"end_char_idx\": 4891, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "doc_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "ref_doc_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2104.01778\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/YuanGongND/ast).\n\n## Usage tips\n\n- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make\nsure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet\nmean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how\nthe authors compute the stats for a downstream dataset.\n- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the\n[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ASTForAudioClassification\nmodel = ASTForAudioClassification.from_pretrained(\"MIT/ast-finetuned-audioset-10-10-0.4593\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        27 |                                         6 |                      4.5 |\n|            2 |                                        12 |                                         6 |                      2   |\n|            4 |                                        21 |                                         8 |                      2.62 |\n|            8 |                                        40 |                                        14 |                      2.86 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with the Audio Spectrogram Transformer."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: ec6dcd84-36d9-4f9c-bc18-11eb71d6baf2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"ec6dcd84-36d9-4f9c-bc18-11eb71d6baf2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8fc5b214-4f01-43d3-8474-dc1d71078291\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"c60a644fba1fa1a6d31884cc274859929434cc364eb0fa31f56b8f8ed0726a90\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40ff3b4e-6223-4539-bc91-aaf28c4d01a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer/\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"00aaae614a4ee9f8b0a250f17e6f06f4b1ef64309acbebdae80f2f8869172477\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3233, \"end_char_idx\": 5926, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "doc_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "ref_doc_id": "8fc5b214-4f01-43d3-8474-dc1d71078291", "chroma:document": "SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ASTForAudioClassification\nmodel = ASTForAudioClassification.from_pretrained(\"MIT/ast-finetuned-audioset-10-10-0.4593\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        27 |                                         6 |                      4.5 |\n|            2 |                                        12 |                                         6 |                      2   |\n|            4 |                                        21 |                                         8 |                      2.62 |\n|            8 |                                        40 |                                        14 |                      2.86 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with the Audio Spectrogram Transformer.\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- A notebook illustrating inference with AST for audio classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/AST).\n- [`ASTForAudioClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n- See also: [Audio classification](../tasks/audio_classification).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ASTConfig\n\n[[autodoc]] ASTConfig\n\n## ASTFeatureExtractor\n\n[[autodoc]] ASTFeatureExtractor\n    - __call__\n\n## ASTModel\n\n[[autodoc]] ASTModel\n    - forward\n\n## ASTForAudioClassification\n\n[[autodoc]] ASTForAudioClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 55bdca4b-6e0a-479a-8897-c0ac019ad574
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"55bdca4b-6e0a-479a-8897-c0ac019ad574\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former/\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f38311c-227d-40e5-91ec-ee742710a0a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former/\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1e405b7a09187cfb7a031c46e362465ca0df8ae5f29ad7dfe8f42b1250880ab2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1b9c3d13-4dd6-44d1-9fab-a5b33f605c30\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b4286353af35aec9c5e32c51e60427e675483b5392132ad419be9ff6b529f1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3409, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "doc_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "ref_doc_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "chroma:document": "# Mask2Former\n\n## Overview\n\nThe Mask2Former model was proposed in [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527) by Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar. Mask2Former is a unified framework for panoptic, instance and semantic segmentation and features significant performance and efficiency improvements over [MaskFormer](maskformer).\n\nThe abstract from the paper is the following:\n\n*Image segmentation groups pixels with different semantics, e.g., category or instance membership. Each choice\nof semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mask2former_architecture.jpg\" alt=\"drawing\" width=\"600\"/>\n\n<small> Mask2Former architecture. Taken from the <a href=\"https://arxiv.org/abs/2112.01527\">original paper.</a> </small>\n\nThis model was contributed by [Shivalika Singh](https://huggingface.co/shivi) and [Alara Dirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/facebookresearch/Mask2Former).\n\n## Usage tips\n\n- Mask2Former uses the same preprocessing and postprocessing steps as [MaskFormer](maskformer). Use [`Mask2FormerImageProcessor`] or [`AutoImageProcessor`] to prepare images and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~Mask2FormerImageProcessor.post_process_semantic_segmentation`] or [`~Mask2FormerImageProcessor.post_process_instance_segmentation`] or [`~Mask2FormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`Mask2FormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mask2Former.\n\n- Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Mask2Former).\n- Scripts for finetuning [`Mask2Former`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1b9c3d13-4dd6-44d1-9fab-a5b33f605c30
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mask2former/", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1b9c3d13-4dd6-44d1-9fab-a5b33f605c30\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former/\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f38311c-227d-40e5-91ec-ee742710a0a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former/\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1e405b7a09187cfb7a031c46e362465ca0df8ae5f29ad7dfe8f42b1250880ab2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"55bdca4b-6e0a-479a-8897-c0ac019ad574\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former/\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"18e4557c3f03019fecefae6c35de9847bfc365a71e7ef6640d0c75713d66f524\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1832, \"end_char_idx\": 4062, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "doc_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "ref_doc_id": "1f38311c-227d-40e5-91ec-ee742710a0a4", "chroma:document": "The original code can be found [here](https://github.com/facebookresearch/Mask2Former).\n\n## Usage tips\n\n- Mask2Former uses the same preprocessing and postprocessing steps as [MaskFormer](maskformer). Use [`Mask2FormerImageProcessor`] or [`AutoImageProcessor`] to prepare images and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~Mask2FormerImageProcessor.post_process_semantic_segmentation`] or [`~Mask2FormerImageProcessor.post_process_instance_segmentation`] or [`~Mask2FormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`Mask2FormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mask2Former.\n\n- Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Mask2Former).\n- Scripts for finetuning [`Mask2Former`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Mask2FormerConfig\n\n[[autodoc]] Mask2FormerConfig\n\n## MaskFormer specific outputs\n\n[[autodoc]] models.mask2former.modeling_mask2former.Mask2FormerModelOutput\n\n[[autodoc]] models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput\n\n## Mask2FormerModel\n\n[[autodoc]] Mask2FormerModel\n    - forward\n\n## Mask2FormerForUniversalSegmentation\n\n[[autodoc]] Mask2FormerForUniversalSegmentation\n    - forward\n\n## Mask2FormerImageProcessor\n\n[[autodoc]] Mask2FormerImageProcessor\n    - preprocess\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 856247ce-1467-4a96-a1b7-8804e881bb32
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"856247ce-1467-4a96-a1b7-8804e881bb32\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer/\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f254130-5f8c-42f3-a424-a35ef400be28\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer/\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2f553e422e3f914165488599c6bf47f8ff836bf1d40842ef08ca56b9b6c3d94c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f05d248b-ad1a-4e66-985e-834db5cfc6db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d8fa86f6e1ff552031aaa60b886f2bb06b82d080b1961a69ca32028a0f30db33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3117, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "doc_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "ref_doc_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "chroma:document": "# PatchTSMixer\n\n## Overview\n\nThe PatchTSMixer model was proposed in [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://arxiv.org/pdf/2306.09364.pdf) by Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong and Jayant Kalagnanam.\n\n\nPatchTSMixer is a lightweight time-series modeling approach based on the MLP-Mixer architecture. In this HuggingFace implementation, we provide PatchTSMixer's capabilities to effortlessly facilitate lightweight mixing across patches, channels, and hidden features for effective multivariate time-series modeling. It also supports various attention mechanisms starting from simple gated attention to more complex self-attention blocks that can be customized accordingly. The model can be pretrained and subsequently used for various downstream tasks such as forecasting, classification and regression.\n\n\nThe abstract from the paper is the following:\n\n*TSMixer is a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules designed for multivariate forecasting and representation learning on patched time series. Our model draws inspiration from the success of MLP-Mixer models in computer vision. We demonstrate the challenges involved in adapting Vision MLP-Mixer for time series and introduce empirically validated components to enhance accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a Hybrid channel modeling approach to effectively handle noisy channel interactions and generalization across diverse datasets, a common challenge in existing patch channel-mixing methods. Additionally, a simple gated attention mechanism is introduced in the backbone to prioritize important features. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X).*\n\nThis model was contributed by [ajati](https://huggingface.co/ajati), [vijaye12](https://huggingface.co/vijaye12), \n[gsinthong](https://huggingface.co/gsinthong), [namctin](https://huggingface.co/namctin),\n[wmgifford](https://huggingface.co/wmgifford), [kashif](https://huggingface.co/kashif).\n\n## Usage example\n\nThe code snippet below shows how to randomly initialize a PatchTSMixer model. The model is compatible with the [Trainer API](../trainer.md).\n\n```python\n\nfrom transformers import PatchTSMixerConfig, PatchTSMixerForPrediction\nfrom transformers import Trainer, TrainingArguments,"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f05d248b-ad1a-4e66-985e-834db5cfc6db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer/", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f05d248b-ad1a-4e66-985e-834db5cfc6db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer/\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8f254130-5f8c-42f3-a424-a35ef400be28\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer/\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2f553e422e3f914165488599c6bf47f8ff836bf1d40842ef08ca56b9b6c3d94c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"856247ce-1467-4a96-a1b7-8804e881bb32\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer/\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"42a730664e9e703e874fed0bc3bac443ad211f2ab402eb3591b9e5bec0c195ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3120, \"end_char_idx\": 4296, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "doc_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "ref_doc_id": "8f254130-5f8c-42f3-a424-a35ef400be28", "chroma:document": "config = PatchTSMixerConfig(context_length = 512, prediction_length = 96)\nmodel = PatchTSMixerForPrediction(config)\ntrainer = Trainer(model=model, args=training_args, \n            train_dataset=train_dataset,\n            eval_dataset=valid_dataset)\ntrainer.train()\nresults = trainer.evaluate(test_dataset)\n```\n\n## Usage tips\n\nThe model can also be used for time series classification and time series regression. See the respective [`PatchTSMixerForTimeSeriesClassification`] and [`PatchTSMixerForRegression`] classes.\n\n## Resources\n\n- A blog post explaining PatchTSMixer in depth can be found [here](https://huggingface.co/blog/patchtsmixer). The blog can also be opened in Google Colab.\n\n## PatchTSMixerConfig\n\n[[autodoc]] PatchTSMixerConfig\n\n\n## PatchTSMixerModel\n\n[[autodoc]] PatchTSMixerModel\n    - forward\n\n\n## PatchTSMixerForPrediction\n\n[[autodoc]] PatchTSMixerForPrediction\n    - forward\n\n\n## PatchTSMixerForTimeSeriesClassification\n\n[[autodoc]] PatchTSMixerForTimeSeriesClassification\n    - forward\n\n\n## PatchTSMixerForPretraining\n\n[[autodoc]] PatchTSMixerForPretraining\n    - forward\n\n\n## PatchTSMixerForRegression\n\n[[autodoc]] PatchTSMixerForRegression\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 0abe423b-c2eb-4095-972d-80ddacbdb8e5
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"0abe423b-c2eb-4095-972d-80ddacbdb8e5\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cf16e5ea-b308-4d82-bd27-462b747b460f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"640543e6-9609-4f07-bead-3efadc87471a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1dd7787d4c5d5d1358e50d40eeb33a44e315385e0003840c427830e461f7a0f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3117, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "ref_doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "chroma:document": "# GPTBigCode\n\n## Overview\n\nThe GPTBigCode model was proposed in [SantaCoder: don't reach for the stars!](https://arxiv.org/abs/2301.03988) by BigCode. The listed authors are: Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo Garc\u00eda del R\u00edo, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra.\n\nThe abstract from the paper is the following:\n\n*The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at [this https URL.](https://huggingface.co/bigcode)*\n\nThe model is an optimized [GPT2 model](https://huggingface.co/docs/transformers/model_doc/gpt2) with support for Multi-Query Attention.\n\n## Implementation details\n\nThe main differences compared to GPT2.\n- Added support for Multi-Query Attention.\n- Use `gelu_pytorch_tanh` instead of classic `gelu`.\n- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).\n- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).\n- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`\n- Cache the attention mask value to avoid recreating it every time.\n- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.\n- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.\n- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 640543e6-9609-4f07-bead-3efadc87471a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"640543e6-9609-4f07-bead-3efadc87471a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cf16e5ea-b308-4d82-bd27-462b747b460f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0abe423b-c2eb-4095-972d-80ddacbdb8e5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"60443551a8bec698603c33b7303e3cb390e9efd90e16cf52146cf04fdf989ebd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f7d5090-7ee9-4574-8ee7-703e43448ca8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0d3cbefe0687d8c467dd8b5332bbc78467db7c65dda39530751c69ff35c60ab5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1483, \"end_char_idx\": 3965, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "ref_doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "chroma:document": "We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at [this https URL.](https://huggingface.co/bigcode)*\n\nThe model is an optimized [GPT2 model](https://huggingface.co/docs/transformers/model_doc/gpt2) with support for Multi-Query Attention.\n\n## Implementation details\n\nThe main differences compared to GPT2.\n- Added support for Multi-Query Attention.\n- Use `gelu_pytorch_tanh` instead of classic `gelu`.\n- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).\n- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).\n- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`\n- Cache the attention mask value to avoid recreating it every time.\n- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.\n- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.\n- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)\n- Use the memory layout (self.num_heads, 3, self.head_dim) instead of `(3, self.num_heads, self.head_dim)` for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible with the original openai-community/gpt2 model).\n\nYou can read more about the optimizations in the [original pull request](https://github.com/huggingface/transformers/pull/22575)\n\n## Combining Starcoder and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5f7d5090-7ee9-4574-8ee7-703e43448ca8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5f7d5090-7ee9-4574-8ee7-703e43448ca8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cf16e5ea-b308-4d82-bd27-462b747b460f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"640543e6-9609-4f07-bead-3efadc87471a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"7da721b38550c1bd7717d317f047ea13002067d150452734ef04e16e4fe47c37\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e5ea052-956b-43d1-a50f-36b35ede398b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc0bfe69c72e629cb05830c35b3b9cb24f0283d0bdaaa27237d12bbde2a8b099\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2271, \"end_char_idx\": 5241, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "ref_doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "chroma:document": "- Added support for Multi-Query Attention.\n- Use `gelu_pytorch_tanh` instead of classic `gelu`.\n- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).\n- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).\n- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`\n- Cache the attention mask value to avoid recreating it every time.\n- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.\n- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.\n- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)\n- Use the memory layout (self.num_heads, 3, self.head_dim) instead of `(3, self.num_heads, self.head_dim)` for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible with the original openai-community/gpt2 model).\n\nYou can read more about the optimizations in the [original pull request](https://github.com/huggingface/transformers/pull/22575)\n\n## Combining Starcoder and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"bigcode/gpt_bigcode-santacoder\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"bigcode/gpt_bigcode-santacoder\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'def hello_world():\\n    print(\"hello world\")\\n\\nif __name__ == \"__main__\":\\n    print(\"hello world\")\\n<|endoftext|>'\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `bigcode/starcoder` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/starcoder-speedup.png\">\n</div>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 7e5ea052-956b-43d1-a50f-36b35ede398b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"7e5ea052-956b-43d1-a50f-36b35ede398b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cf16e5ea-b308-4d82-bd27-462b747b460f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"dee674aa00a8ee30f0f0a3a8c1877973b3ca3ea6bce3edae08a3a2a08b5df529\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f7d5090-7ee9-4574-8ee7-703e43448ca8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode/\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e1fc7e37ce1e93eeed7e56705f61dda3588ad7a4b0a9545c19b35da4419a782b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3690, \"end_char_idx\": 5631, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "ref_doc_id": "cf16e5ea-b308-4d82-bd27-462b747b460f", "chroma:document": "```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"bigcode/gpt_bigcode-santacoder\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"bigcode/gpt_bigcode-santacoder\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'def hello_world():\\n    print(\"hello world\")\\n\\nif __name__ == \"__main__\":\\n    print(\"hello world\")\\n<|endoftext|>'\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `bigcode/starcoder` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/starcoder-speedup.png\">\n</div>\n\n\n## GPTBigCodeConfig\n\n[[autodoc]] GPTBigCodeConfig\n\n## GPTBigCodeModel\n\n[[autodoc]] GPTBigCodeModel\n    - forward\n\n## GPTBigCodeForCausalLM\n\n[[autodoc]] GPTBigCodeForCausalLM\n    - forward\n\n## GPTBigCodeForSequenceClassification\n\n[[autodoc]] GPTBigCodeForSequenceClassification\n    - forward\n\n## GPTBigCodeForTokenClassification\n\n[[autodoc]] GPTBigCodeForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 150d2a9b-e139-4622-b166-17fb97910339
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nystromformer/", "title": "Nystr\u00f6mformer", "tokens": 796, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"150d2a9b-e139-4622-b166-17fb97910339\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nystromformer/\", \"title\": \"Nystr\\u00f6mformer\", \"tokens\": 796, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98051f69-9a2a-4bd6-9c66-4a2042b35beb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nystromformer/\", \"title\": \"Nystr\\u00f6mformer\", \"tokens\": 796, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"79839e0f1f69b61c039f942441047f9c7815d2f119980341a469d0458d1fb3d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2705, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98051f69-9a2a-4bd6-9c66-4a2042b35beb", "doc_id": "98051f69-9a2a-4bd6-9c66-4a2042b35beb", "ref_doc_id": "98051f69-9a2a-4bd6-9c66-4a2042b35beb", "chroma:document": "# Nystr\u00f6mformer\n\n## Overview\n\nThe Nystr\u00f6mformer model was proposed in [*Nystr\u00f6mformer: A Nystr\u00f6m-Based Algorithm for Approximating Self-Attention*](https://arxiv.org/abs/2102.03902) by Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn\nFung, Yin Li, and Vikas Singh.\n\nThe abstract from the paper is the following:\n\n*Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component\nthat drives the impressive performance of Transformers is the self-attention mechanism that encodes the influence or\ndependence of other tokens on each specific token. While beneficial, the quadratic complexity of self-attention on the\ninput sequence length has limited its application to longer sequences -- a topic being actively studied in the\ncommunity. To address this limitation, we propose Nystr\u00f6mformer -- a model that exhibits favorable scalability as a\nfunction of sequence length. Our idea is based on adapting the Nystr\u00f6m method to approximate standard self-attention\nwith O(n) complexity. The scalability of Nystr\u00f6mformer enables application to longer sequences with thousands of\ntokens. We perform evaluations on multiple downstream tasks on the GLUE benchmark and IMDB reviews with standard\nsequence length, and find that our Nystr\u00f6mformer performs comparably, or in a few cases, even slightly better, than\nstandard self-attention. On longer sequence tasks in the Long Range Arena (LRA) benchmark, Nystr\u00f6mformer performs\nfavorably relative to other efficient self-attention methods. Our code is available at this https URL.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/Nystromformer).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## NystromformerConfig\n\n[[autodoc]] NystromformerConfig\n\n## NystromformerModel\n\n[[autodoc]] NystromformerModel\n    - forward\n\n## NystromformerForMaskedLM\n\n[[autodoc]] NystromformerForMaskedLM\n    - forward\n\n## NystromformerForSequenceClassification\n\n[[autodoc]] NystromformerForSequenceClassification\n    - forward\n\n## NystromformerForMultipleChoice\n\n[[autodoc]] NystromformerForMultipleChoice\n    - forward\n\n## NystromformerForTokenClassification\n\n[[autodoc]] NystromformerForTokenClassification\n    - forward\n\n## NystromformerForQuestionAnswering\n\n[[autodoc]] NystromformerForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 51c98a32-bc2d-4536-b21a-dc0bd19e5b13
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/glpn/", "title": "GLPN", "tokens": 895, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"51c98a32-bc2d-4536-b21a-dc0bd19e5b13\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/glpn/\", \"title\": \"GLPN\", \"tokens\": 895, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9cd5b097-c0ee-4633-9938-bb8949124f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/glpn/\", \"title\": \"GLPN\", \"tokens\": 895, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"414b5f3674d24a0aabce2e5beabfa71ab84e94e3423c05fecd044cda0b06ac3d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9cd5b097-c0ee-4633-9938-bb8949124f82", "doc_id": "9cd5b097-c0ee-4633-9938-bb8949124f82", "ref_doc_id": "9cd5b097-c0ee-4633-9938-bb8949124f82", "chroma:document": "# GLPN\n\n<Tip>\n\nThis is a recently introduced model so the API hasn't been tested extensively. There may be some bugs or slight\nbreaking changes to fix it in the future. If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title).\n\n</Tip>\n\n## Overview\n\nThe GLPN model was proposed in [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436)  by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim.\nGLPN combines [SegFormer](segformer)'s hierarchical mix-Transformer with a lightweight decoder for monocular depth estimation. The proposed decoder shows better performance than the previously proposed decoders, with considerably\nless computational complexity.\n\nThe abstract from the paper is the following:\n\n*Depth estimation from a single image is an important task that can be applied to various fields in computer vision, and has grown rapidly with the development of convolutional neural networks. In this paper, we propose a novel structure and training strategy for monocular depth estimation to further improve the prediction accuracy of the network. We deploy a hierarchical transformer encoder to capture and convey the global context, and design a lightweight yet powerful decoder to generate an estimated depth map while considering local connectivity. By constructing connected paths between multi-scale local features and the global decoding stream with our proposed selective feature fusion module, the network can integrate both representations and recover fine details. In addition, the proposed decoder shows better performance than the previously proposed decoders, with considerably less computational complexity. Furthermore, we improve the depth-specific augmentation method by utilizing an important observation in depth estimation to enhance the model. Our network achieves state-of-the-art performance over the challenging depth dataset NYU Depth V2. Extensive experiments have been conducted to validate and show the effectiveness of the proposed approach. Finally, our model shows better generalisation ability and robustness than other comparative models.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Summary of the approach. Taken from the <a href=\"https://arxiv.org/abs/2201.07436\" target=\"_blank\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/vinvino02/GLPDepth).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GLPN.\n\n- Demo notebooks for [`GLPNForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GLPN).\n- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)\n\n## GLPNConfig\n\n[[autodoc]] GLPNConfig\n\n## GLPNFeatureExtractor\n\n[[autodoc]] GLPNFeatureExtractor\n    - __call__\n\n## GLPNImageProcessor\n\n[[autodoc]] GLPNImageProcessor\n    - preprocess\n\n## GLPNModel\n\n[[autodoc]] GLPNModel\n    - forward\n\n## GLPNForDepthEstimation\n\n[[autodoc]] GLPNForDepthEstimation\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1a395ab8-4203-4a51-9777-4200ee2f7c6e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gemma/", "title": "Gemma", "tokens": 661, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1a395ab8-4203-4a51-9777-4200ee2f7c6e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma/\", \"title\": \"Gemma\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a2f21922-1938-4f7a-9c7e-460ddb249444\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma/\", \"title\": \"Gemma\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"acc214dff7a9ff1d4d6398eba07c40fd18a349bc40bb24f6128a87dc81ae78db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1970, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a2f21922-1938-4f7a-9c7e-460ddb249444", "doc_id": "a2f21922-1938-4f7a-9c7e-460ddb249444", "ref_doc_id": "a2f21922-1938-4f7a-9c7e-460ddb249444", "chroma:document": "# Gemma\n\n## Overview\n\nThe Gemma model was proposed in [Gemma: Open Models Based on Gemini Technology and Research](https://blog.google/technology/developers/gemma-open-models/) by Gemma Team, Google.\nGemma models are trained on 6T tokens, and released with 2 versions, 2b and 7b.\n\nThe abstract from the paper is the following:\n\n*This work introduces Gemma, a new family of open language models demonstrating strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of our model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations*\n\nTips:\n\n- The original checkpoints can be converted using the conversion script `src/transformers/models/gemma/convert_gemma_weights_to_hf.py` \n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Younes Belkada](https://huggingface.co/ybelkada), [Sanchit Gandhi](https://huggingface.co/sanchit-gandhi), [Pedro Cuenca](https://huggingface.co/pcuenq).\n\n\n## GemmaConfig\n\n[[autodoc]] GemmaConfig\n\n## GemmaTokenizer\n\n[[autodoc]] GemmaTokenizer\n\n\n## GemmaTokenizerFast\n\n[[autodoc]] GemmaTokenizerFast\n\n## GemmaModel\n\n[[autodoc]] GemmaModel\n    - forward\n\n## GemmaForCausalLM\n\n[[autodoc]] GemmaForCausalLM\n    - forward\n\n## GemmaForSequenceClassification\n\n[[autodoc]] GemmaForSequenceClassification\n    - forward\n\n## GemmaForTokenClassification\n\n[[autodoc]] GemmaForTokenClassification\n    - forward\n\n## FlaxGemmaModel\n\n[[autodoc]] FlaxGemmaModel\n    - __call__\n\n## FlaxGemmaForCausalLM\n\n[[autodoc]] FlaxGemmaForCausalLM\n    - __call__"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 08a9b906-fa64-4144-a9fe-9ffa9879a021
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wavlm/", "title": "WavLM", "tokens": 859, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"08a9b906-fa64-4144-a9fe-9ffa9879a021\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wavlm/\", \"title\": \"WavLM\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4375e553-9e0d-4824-8b64-31a21aec6610\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wavlm/\", \"title\": \"WavLM\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"76a15cd30edcb100bdc511288d2eea3c75c058998f9065c2d4b4603384847b22\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2851, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4375e553-9e0d-4824-8b64-31a21aec6610", "doc_id": "4375e553-9e0d-4824-8b64-31a21aec6610", "ref_doc_id": "4375e553-9e0d-4824-8b64-31a21aec6610", "chroma:document": "# WavLM\n\n## Overview\n\nThe WavLM model was proposed in [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://arxiv.org/abs/2110.13900) by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen,\nJinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu,\nMichael Zeng, Furu Wei.\n\nThe abstract from the paper is the following:\n\n*Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been\nattempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker\nidentity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is\nchallenging. In this paper, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks.\nWavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity\npreservation. We first equip the Transformer structure with gated relative position bias to improve its capability on\nrecognition tasks. For better speaker discrimination, we propose an utterance mixing training strategy, where\nadditional overlapped utterances are created unsupervisedly and incorporated during model training. Lastly, we scale up\nthe training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB\nbenchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.*\n\nRelevant checkpoints can be found under https://huggingface.co/models?other=wavlm.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/microsoft/unilm/tree/master/wavlm).\n\n## Usage tips\n\n- WavLM is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. Please use\n  [`Wav2Vec2Processor`] for the feature extraction.\n- WavLM model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n- WavLM performs especially well on speaker verification, speaker identification, and speaker diarization tasks.\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## WavLMConfig\n\n[[autodoc]] WavLMConfig\n\n## WavLMModel\n\n[[autodoc]] WavLMModel\n    - forward\n\n## WavLMForCTC\n\n[[autodoc]] WavLMForCTC\n    - forward\n\n## WavLMForSequenceClassification\n\n[[autodoc]] WavLMForSequenceClassification\n    - forward\n\n## WavLMForAudioFrameClassification\n\n[[autodoc]] WavLMForAudioFrameClassification\n    - forward\n\n## WavLMForXVector\n\n[[autodoc]] WavLMForXVector\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: f27ffbf5-b218-403a-a997-07f73a950c42
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/unispeech-sat/", "title": "UniSpeech-SAT", "tokens": 891, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"f27ffbf5-b218-403a-a997-07f73a950c42\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech-sat/\", \"title\": \"UniSpeech-SAT\", \"tokens\": 891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b141613c-b743-4fa7-88d1-f3f3e7bb41d8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech-sat/\", \"title\": \"UniSpeech-SAT\", \"tokens\": 891, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"44bd08737e3b53a1307a999b0ebbf5a66ec3d6e576354ebd27634ecdfc9f1d34\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3262, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b141613c-b743-4fa7-88d1-f3f3e7bb41d8", "doc_id": "b141613c-b743-4fa7-88d1-f3f3e7bb41d8", "ref_doc_id": "b141613c-b743-4fa7-88d1-f3f3e7bb41d8", "chroma:document": "# UniSpeech-SAT\n\n## Overview\n\nThe UniSpeech-SAT model was proposed in [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware\nPre-Training](https://arxiv.org/abs/2110.05752) by Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen,\nShujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu .\n\nThe abstract from the paper is the following:\n\n*Self-supervised learning (SSL) is a long-standing goal for speech processing, since it utilizes large-scale unlabeled\ndata and avoids extensive human labeling. Recent years witness great successes in applying self-supervised learning in\nspeech recognition, while limited exploration was attempted in applying SSL for modeling speaker characteristics. In\nthis paper, we aim to improve the existing SSL framework for speaker representation learning. Two methods are\nintroduced for enhancing the unsupervised speaker information extraction. First, we apply the multi-task learning to\nthe current SSL framework, where we integrate the utterance-wise contrastive loss with the SSL objective function.\nSecond, for better speaker discrimination, we propose an utterance mixing strategy for data augmentation, where\nadditional overlapped utterances are created unsupervisedly and incorporate during training. We integrate the proposed\nmethods into the HuBERT framework. Experiment results on SUPERB benchmark show that the proposed system achieves\nstate-of-the-art performance in universal representation learning, especially for speaker identification oriented\ntasks. An ablation study is performed verifying the efficacy of each proposed method. Finally, we scale up training\ndataset to 94 thousand hours public audio data and achieve further performance improvement in all SUPERB tasks.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/microsoft/UniSpeech/tree/main/UniSpeech-SAT).\n\n## Usage tips\n\n- UniSpeechSat is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n  Please use [`Wav2Vec2Processor`] for the feature extraction.\n- UniSpeechSat model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2CTCTokenizer`].\n- UniSpeechSat performs especially well on speaker verification, speaker identification, and speaker diarization tasks.\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## UniSpeechSatConfig\n\n[[autodoc]] UniSpeechSatConfig\n\n## UniSpeechSat specific outputs\n\n[[autodoc]] models.unispeech_sat.modeling_unispeech_sat.UniSpeechSatForPreTrainingOutput\n\n## UniSpeechSatModel\n\n[[autodoc]] UniSpeechSatModel\n    - forward\n\n## UniSpeechSatForCTC\n\n[[autodoc]] UniSpeechSatForCTC\n    - forward\n\n## UniSpeechSatForSequenceClassification\n\n[[autodoc]] UniSpeechSatForSequenceClassification\n    - forward\n\n## UniSpeechSatForAudioFrameClassification\n\n[[autodoc]] UniSpeechSatForAudioFrameClassification\n    - forward\n\n## UniSpeechSatForXVector\n\n[[autodoc]] UniSpeechSatForXVector\n    - forward\n\n## UniSpeechSatForPreTraining\n\n[[autodoc]] UniSpeechSatForPreTraining\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 90d68094-7547-44be-91ee-15c9a0a7dbfb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"90d68094-7547-44be-91ee-15c9a0a7dbfb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr/\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14f4911d-07ea-4cae-a49c-bca7b1b308c8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr/\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"56b9377c1e5aea4a1be7e9db377bc5b4cad9af43c2c12af715b989a47c124168\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3e230e44-5da9-400c-a997-33ec7f426e22\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"21851b0c561b8518650206f11e00571ae7bd1cd5fd205a0df870c75ce5f692e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "doc_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "ref_doc_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "chroma:document": "# DPR\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=dpr\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-dpr-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/dpr-question_encoder-bert-base-multilingual\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nDense Passage Retrieval (DPR) is a set of tools and models for state-of-the-art open-domain Q&A research. It was\nintroduced in [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) by\nVladimir Karpukhin, Barlas O\u011fuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.\n\nThe abstract from the paper is the following:\n\n*Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional\nsparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can\nbe practically implemented using dense representations alone, where embeddings are learned from a small number of\nquestions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,\nour dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage\nretrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA\nbenchmarks.*\n\nThis model was contributed by [lhoestq](https://huggingface.co/lhoestq). The original code can be found [here](https://github.com/facebookresearch/DPR).\n\n## Usage tips\n\n- DPR consists in three models:\n\n    * Question encoder: encode questions as vectors\n    * Context encoder: encode contexts as vectors\n    * Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 3e230e44-5da9-400c-a997-33ec7f426e22
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpr/", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"3e230e44-5da9-400c-a997-33ec7f426e22\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr/\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"14f4911d-07ea-4cae-a49c-bca7b1b308c8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr/\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"56b9377c1e5aea4a1be7e9db377bc5b4cad9af43c2c12af715b989a47c124168\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90d68094-7547-44be-91ee-15c9a0a7dbfb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr/\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9af9dcdb1ead3dda4a4c70f5268acfa4ebdea600f05c9d2603d2e94ee738ffd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 512, \"end_char_idx\": 3067, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "doc_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "ref_doc_id": "14f4911d-07ea-4cae-a49c-bca7b1b308c8", "chroma:document": "It was\nintroduced in [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) by\nVladimir Karpukhin, Barlas O\u011fuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.\n\nThe abstract from the paper is the following:\n\n*Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional\nsparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can\nbe practically implemented using dense representations alone, where embeddings are learned from a small number of\nquestions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,\nour dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage\nretrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA\nbenchmarks.*\n\nThis model was contributed by [lhoestq](https://huggingface.co/lhoestq). The original code can be found [here](https://github.com/facebookresearch/DPR).\n\n## Usage tips\n\n- DPR consists in three models:\n\n    * Question encoder: encode questions as vectors\n    * Context encoder: encode contexts as vectors\n    * Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question).\n\n## DPRConfig\n\n[[autodoc]] DPRConfig\n\n## DPRContextEncoderTokenizer\n\n[[autodoc]] DPRContextEncoderTokenizer\n\n## DPRContextEncoderTokenizerFast\n\n[[autodoc]] DPRContextEncoderTokenizerFast\n\n## DPRQuestionEncoderTokenizer\n\n[[autodoc]] DPRQuestionEncoderTokenizer\n\n## DPRQuestionEncoderTokenizerFast\n\n[[autodoc]] DPRQuestionEncoderTokenizerFast\n\n## DPRReaderTokenizer\n\n[[autodoc]] DPRReaderTokenizer\n\n## DPRReaderTokenizerFast\n\n[[autodoc]] DPRReaderTokenizerFast\n\n## DPR specific outputs\n\n[[autodoc]] models.dpr.modeling_dpr.DPRContextEncoderOutput\n\n[[autodoc]] models.dpr.modeling_dpr.DPRQuestionEncoderOutput\n\n[[autodoc]] models.dpr.modeling_dpr.DPRReaderOutput\n\n<frameworkcontent>\n<pt>\n\n## DPRContextEncoder\n\n[[autodoc]] DPRContextEncoder\n    - forward\n\n## DPRQuestionEncoder\n\n[[autodoc]] DPRQuestionEncoder\n    - forward\n\n## DPRReader\n\n[[autodoc]] DPRReader\n    - forward\n\n</pt>\n<tf>\n\n## TFDPRContextEncoder\n\n[[autodoc]] TFDPRContextEncoder\n    - call\n\n## TFDPRQuestionEncoder\n\n[[autodoc]] TFDPRQuestionEncoder\n    - call\n\n## TFDPRReader\n\n[[autodoc]] TFDPRReader\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: da9d2e2c-b742-4e68-8d56-6f802bbf8536
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/realm/", "title": "REALM", "tokens": 856, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"da9d2e2c-b742-4e68-8d56-6f802bbf8536\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/realm/\", \"title\": \"REALM\", \"tokens\": 856, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"04ce72f2-eab1-44d9-bc35-0661dbb866f4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/realm/\", \"title\": \"REALM\", \"tokens\": 856, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8cd9deb3f768c6ce0228ef2d2d000faac5d5b99296374d67c7aa03a71b8a4c30\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "04ce72f2-eab1-44d9-bc35-0661dbb866f4", "doc_id": "04ce72f2-eab1-44d9-bc35-0661dbb866f4", "ref_doc_id": "04ce72f2-eab1-44d9-bc35-0661dbb866f4", "chroma:document": "# REALM\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe REALM model was proposed in [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909) by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang. It's a\nretrieval-augmented language model that firstly retrieves documents from a textual knowledge corpus and then\nutilizes retrieved documents to process question answering tasks.\n\nThe abstract from the paper is the following:\n\n*Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks\nsuch as question answering. However, this knowledge is stored implicitly in the parameters of a neural network,\nrequiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we\naugment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend\nover documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the\nfirst time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language\nmodeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We\ndemonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the\nchallenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both\nexplicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous\nmethods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as\ninterpretability and modularity.*\n\nThis model was contributed by [qqaatw](https://huggingface.co/qqaatw). The original code can be found\n[here](https://github.com/google-research/language/tree/master/language/realm).\n\n## RealmConfig\n\n[[autodoc]] RealmConfig\n\n## RealmTokenizer\n\n[[autodoc]] RealmTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n    - batch_encode_candidates\n\n## RealmTokenizerFast\n\n[[autodoc]] RealmTokenizerFast\n    - batch_encode_candidates\n\n## RealmRetriever\n\n[[autodoc]] RealmRetriever\n\n## RealmEmbedder\n\n[[autodoc]] RealmEmbedder\n    - forward\n\n## RealmScorer\n\n[[autodoc]] RealmScorer\n    - forward\n\n## RealmKnowledgeAugEncoder\n\n[[autodoc]] RealmKnowledgeAugEncoder\n    - forward\n\n## RealmReader\n\n[[autodoc]] RealmReader\n    - forward\n\n## RealmForOpenQA\n\n[[autodoc]] RealmForOpenQA\n    - block_embedding_to\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: d589a30b-957f-4f41-ad94-38f72e920d47
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"d589a30b-957f-4f41-ad94-38f72e920d47\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae4bb8-566e-4c73-a06c-2b42b40650ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"98b923fb-63d8-4301-bab1-898028454305\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5afae6ed9e4f976602b8ef2c2b78cf83f4a5d765f2c27de0b3f692d07bcc6e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3354, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "ref_doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "chroma:document": "# Video-LLaVA\n\n## Overview\n\nVideo-LLaVa is an open-source multimodal LLM trained by fine-tuning LlamA/Vicuna on multimodal instruction-following data generated by Llava1.5 and VideChat. It is an auto-regressive language model, based on the transformer architecture. Video-LLaVa unifies visual representations to the language feature space, and enables an LLM to perform visual reasoning capabilities on both images and videos simultaneously.\n\n\nThe Video-LLaVA model was proposed in [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://arxiv.org/abs/2311.10122) by Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munang Ning, Peng Jin, Li Yuan.\n\nThe abstract from the paper is the following:\n\n*The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in\nvisual-language understanding. Most existing approaches\nencode images and videos into separate feature spaces,\nwhich are then fed as inputs to large language models.\nHowever, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it\nbecomes challenging for a Large Language Model (LLM)\nto learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA,\nwhich learns from a mixed dataset of images and videos,\nmutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4\nimage benchmark toolkits. Additionally, our Video-LLaVA\nalso outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%,\nand 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that\nVideo-LLaVA mutually benefits images and videos within\na unified visual representation, outperforming models designed specifically for images or videos. We aim for this\nwork to provide modest insights into the multi-modal inputs\nfor the LLM*\n\n## Usage tips:\n\n- We advise users to use padding_side=\"left\" when computing batched generation as it leads to more accurate results. Simply make sure to call processor.tokenizer.padding_side = \"left\" before generating.\n\n- Note the model has not been explicitly trained to process multiple images/videos in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- Note that the video inputs should have exactly 8 frames at the input, since the models were trained in that setting. \n\nThis model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/PKU-YuanGroup/Video-LLaVA).\n\n\n## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 98b923fb-63d8-4301-bab1-898028454305
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"98b923fb-63d8-4301-bab1-898028454305\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae4bb8-566e-4c73-a06c-2b42b40650ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d589a30b-957f-4f41-ad94-38f72e920d47\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"004f358524466c61ef5d6ac4d7a4bfdd4b20f6d6e9cbe32ea8cdc3104e3550a2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4245e9a0-d00b-48f3-a00d-c12e253bbe6e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9855767cc8352d9cd036b2b8755bf1d71179ea38a63cb4113a60669da8d8556\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2815, \"end_char_idx\": 5679, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "ref_doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "chroma:document": "## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode.\n    Returns:\n        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).\n    '''\n    frames = []\n    container.seek(0)\n    start_index = indices[0]\n    end_index = indices[-1]\n    for i, frame in enumerate(container.decode(video=0)):\n        if i > end_index:\n            break\n        if i >= start_index and i in indices:\n            frames.append(frame)\n    return np.stack([x.to_ndarray(format=\"rgb24\") for x in frames])\n\n# Load the model in half-precision\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = VideoLlavaProcessor.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\")\n\n# Load the video as an np.arrau, sampling uniformly 8 frames\nvideo_path = hf_hub_download(repo_id=\"raushan-testing-hf/videos-test\", filename=\"sample_demo_1.mp4\", repo_type=\"dataset\")\ncontainer = av.open(video_path)\ntotal_frames = container.streams.video[0].frames\nindices = np.arange(0, total_frames, total_frames / 8).astype(int)\nvideo = read_video_pyav(container, indices)\n\n# For better results, we recommend to prompt the model in the following format\nprompt = \"USER: <video>\\nWhy is this funny? ASSISTANT:\"\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```\n\nFor multiple turns conversation change the prompt format to:\n\n```bash\n\"USER: <video>\\nWhat do you see in this video? ASSISTANT: A baby reading a book. USER: Why is the it funny? ASSISTANT:\"\n```\n\n### Mixed Media Mode\n\nThe model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: \n\n```python\nfrom PIL import Image\nimport requests\n\n# Generate from image and video mixed inputs\n# Load and image and write a new prompt\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"USER: <image>\\nHow many cats are there in the image? ASSISTANT: There are two cats. USER: <video>\\nWhy is this video funny?"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4245e9a0-d00b-48f3-a00d-c12e253bbe6e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4245e9a0-d00b-48f3-a00d-c12e253bbe6e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae4bb8-566e-4c73-a06c-2b42b40650ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"98b923fb-63d8-4301-bab1-898028454305\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"1c09b739643997ff775bdaa294e78304d2f40f3838f2d65cda4f4a9029aab41b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"72686aed-d61f-4e94-aa52-a089dbd60eb1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0958f432101801dca225bcf8728ea85bc59940e6ac73b194fa35ee49e6ced23\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4574, \"end_char_idx\": 6941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "ref_doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "chroma:document": "ASSISTANT:\"\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```\n\nFor multiple turns conversation change the prompt format to:\n\n```bash\n\"USER: <video>\\nWhat do you see in this video? ASSISTANT: A baby reading a book. USER: Why is the it funny? ASSISTANT:\"\n```\n\n### Mixed Media Mode\n\nThe model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: \n\n```python\nfrom PIL import Image\nimport requests\n\n# Generate from image and video mixed inputs\n# Load and image and write a new prompt\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"USER: <image>\\nHow many cats are there in the image? ASSISTANT: There are two cats. USER: <video>\\nWhy is this video funny? ASSISTANT:\"\n\ninputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors=\"pt\")\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_length=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes for memory efficiency\n\nThe model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. his allows for efficient deployment on resource-constrained cases. \n\nFirst make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:\n\n\n```python\nfrom transformers import VideoLlavaForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 72686aed-d61f-4e94-aa52-a089dbd60eb1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava/", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"72686aed-d61f-4e94-aa52-a089dbd60eb1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"59ae4bb8-566e-4c73-a06c-2b42b40650ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5605ee185aaf606f4ebb9ab6d579fa8ea7269908854a67eb699fbc87788925f5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4245e9a0-d00b-48f3-a00d-c12e253bbe6e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava/\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bdafe818e6c7388a2c943ff3741b91301dfba9d7f7b77914ff0deefb3a52c9e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5609, \"end_char_idx\": 8309, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "ref_doc_id": "59ae4bb8-566e-4c73-a06c-2b42b40650ee", "chroma:document": "ASSISTANT: There are two cats. USER: <video>\\nWhy is this video funny? ASSISTANT:\"\n\ninputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors=\"pt\")\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_length=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes for memory efficiency\n\nThe model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. his allows for efficient deployment on resource-constrained cases. \n\nFirst make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:\n\n\n```python\nfrom transformers import VideoLlavaForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```\n\n\n### Flash-Attention 2 to speed-up generation\n\nAdditionally, we can greatly speed-up model inference by using [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.\n\nTo load and run a model using Flash Attention-2, simply add `attn_implementation=\"flash_attention_2\"` when loading the model as follows:\n\n```python\nfrom transformers import VideoLlavaForConditionalGeneration\n\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\n    \"LanguageBind/Video-LLaVA-7B-hf\", \n    torch_dtype=torch.float16, \n    attn_implementation=\"flash_attention_2\",\n).to(0)\n```\n\n\n## VideoLlavaConfig\n\n[[autodoc]] VideoLlavaConfig\n\n## VideoLlavaImageProcessor\n\n[[autodoc]] VideoLlavaImageProcessor\n\n## VideoLlavaProcessor\n\n[[autodoc]] VideoLlavaProcessor\n\n## VideoLlavaForConditionalGeneration\n\n[[autodoc]] VideoLlavaForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 80328af8-d89e-470f-9f46-75db5b5bfb93
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"80328af8-d89e-470f-9f46-75db5b5bfb93\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str/\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str/\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a2188600d847f2ffd6505a575b3783b5c119c24da1562b50a0fde6d51ecaa49a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cdfb2f72-be24-47b6-94ed-0829bb945d1a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"effd02f5291e1b8394a3f26c9c38728c1bce69dcc2f46c046297a53b881f3565\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3236, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "doc_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "ref_doc_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "chroma:document": "# MGP-STR\n\n## Overview\n\nThe MGP-STR model was proposed in [Multi-Granularity Prediction for Scene Text Recognition](https://arxiv.org/abs/2209.03592) by Peng Wang, Cheng Da, and Cong Yao. MGP-STR is a conceptually **simple** yet **powerful** vision Scene Text Recognition (STR) model, which is built upon the [Vision Transformer (ViT)](vit). To integrate linguistic knowledge, Multi-Granularity Prediction (MGP) strategy is proposed to inject information from the language modality into the model in an implicit way.\n\nThe abstract from the paper is the following:\n\n*Scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this challenging problem, numerous innovative methods have been successively proposed and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet powerful vision STR model, which is built upon ViT and outperforms previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely-used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. The resultant algorithm (termed MGP-STR) is able to push the performance envelop of STR to an even higher level. Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mgp_str_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> MGP-STR architecture. Taken from the <a href=\"https://arxiv.org/abs/2209.03592\">original paper</a>. </small>\n\nMGP-STR is trained on two synthetic datasets [MJSynth]((http://www.robots.ox.ac.uk/~vgg/data/text/)) (MJ) and [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE).\nThis model was contributed by [yuekun](https://huggingface.co/yuekun). The original code can be found [here](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).\n\n## Inference example\n\n[`MgpstrModel`] accepts images as input and generates three types of predictions, which represent textual information at different granularities.\nThe three types of predictions are fused to give the final prediction result.\n\nThe [`ViTImageProcessor`] class is responsible for preprocessing the input image and\n[`MgpstrTokenizer`] decodes the generated character tokens to the target string. The\n[`MgpstrProcessor`] wraps [`ViTImageProcessor`] and [`MgpstrTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cdfb2f72-be24-47b6-94ed-0829bb945d1a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str/", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cdfb2f72-be24-47b6-94ed-0829bb945d1a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str/\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str/\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a2188600d847f2ffd6505a575b3783b5c119c24da1562b50a0fde6d51ecaa49a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"80328af8-d89e-470f-9f46-75db5b5bfb93\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str/\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"443b2ba9d34df9b05a0e51903aa0126c6492ba361ff389d771479c58e03e9ea0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1691, \"end_char_idx\": 4290, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "doc_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "ref_doc_id": "deb21e82-bd1e-4beb-a2ca-c6e2b76bfa6d", "chroma:document": "Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mgp_str_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> MGP-STR architecture. Taken from the <a href=\"https://arxiv.org/abs/2209.03592\">original paper</a>. </small>\n\nMGP-STR is trained on two synthetic datasets [MJSynth]((http://www.robots.ox.ac.uk/~vgg/data/text/)) (MJ) and [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE).\nThis model was contributed by [yuekun](https://huggingface.co/yuekun). The original code can be found [here](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).\n\n## Inference example\n\n[`MgpstrModel`] accepts images as input and generates three types of predictions, which represent textual information at different granularities.\nThe three types of predictions are fused to give the final prediction result.\n\nThe [`ViTImageProcessor`] class is responsible for preprocessing the input image and\n[`MgpstrTokenizer`] decodes the generated character tokens to the target string. The\n[`MgpstrProcessor`] wraps [`ViTImageProcessor`] and [`MgpstrTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Optical Character Recognition (OCR)\n\n```py\n>>> from transformers import MgpstrProcessor, MgpstrForSceneTextRecognition\n>>> import requests\n>>> from PIL import Image\n\n>>> processor = MgpstrProcessor.from_pretrained('alibaba-damo/mgp-str-base')\n>>> model = MgpstrForSceneTextRecognition.from_pretrained('alibaba-damo/mgp-str-base')\n\n>>> # load image from the IIIT-5k dataset\n>>> url = \"https://i.postimg.cc/ZKwLg2Gw/367-14.png\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n\n>>> pixel_values = processor(images=image, return_tensors=\"pt\").pixel_values\n>>> outputs = model(pixel_values)\n\n>>> generated_text = processor.batch_decode(outputs.logits)['generated_text']\n```\n\n## MgpstrConfig\n\n[[autodoc]] MgpstrConfig\n\n## MgpstrTokenizer\n\n[[autodoc]] MgpstrTokenizer\n    - save_vocabulary\n\n## MgpstrProcessor\n\n[[autodoc]] MgpstrProcessor\n    - __call__\n    - batch_decode\n\n## MgpstrModel\n\n[[autodoc]] MgpstrModel\n    - forward\n\n## MgpstrForSceneTextRecognition\n\n[[autodoc]] MgpstrForSceneTextRecognition\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 682d3aa8-4e0f-4cb7-9de4-da8a9693ced7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"682d3aa8-4e0f-4cb7-9de4-da8a9693ced7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega/\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75e0d8e2-5719-4d8f-a178-d1d8f0f79788\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega/\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3d9e40b337b6e30e0b8e81ca60c733b8e0772d11332e013ad9bff337e9a34b5f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e24bd266-cd8b-43da-82e1-c9ea30baca85\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5efce74adbd19bf6f2effa81a4a3e3741e2f9e898c475ccbda2f5df62728a6f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3237, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "doc_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "ref_doc_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "chroma:document": "# MEGA\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe MEGA model was proposed in [Mega: Moving Average Equipped Gated Attention](https://arxiv.org/abs/2209.10655) by Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.\nMEGA proposes a new approach to self-attention with each encoder layer having a multi-headed exponential moving average in addition to a single head of standard dot-product attention, giving the attention mechanism\nstronger positional biases. This allows MEGA to perform competitively to Transformers on standard benchmarks including LRA\nwhile also having significantly fewer parameters. MEGA's compute efficiency allows it to scale to very long sequences, making it an\nattractive option for long-document NLP tasks.\n\nThe abstract from the paper is the following:\n\n *The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of Mega that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that Mega achieves significant improvements over other sequence models, including variants of Transformers and recent state space models. *\n\nThis model was contributed by [mnaylor](https://huggingface.co/mnaylor).\nThe original code can be found [here](https://github.com/facebookresearch/mega).\n\n\n## Usage tips\n\n- MEGA can perform quite well with relatively few parameters. See Appendix D in the MEGA paper for examples of architectural specs which perform well in various settings. If using MEGA as a decoder, be sure to set `bidirectional=False` to avoid errors with default bidirectional.\n- Mega-chunk is a variant of mega that reduces time and spaces complexity from quadratic to linear. Utilize chunking with MegaConfig.use_chunking and control chunk size with MegaConfig.chunk_size\n\n\n## Implementation Notes\n\n- The original implementation of MEGA had an inconsistent expectation of attention masks for padding and causal self-attention between the softmax attention and Laplace/squared ReLU method. This implementation addresses that inconsistency.\n- The original implementation did not include token type embeddings; this implementation adds support for these, with the option controlled by MegaConfig.add_token_type_embeddings"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: e24bd266-cd8b-43da-82e1-c9ea30baca85
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mega/", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"e24bd266-cd8b-43da-82e1-c9ea30baca85\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega/\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75e0d8e2-5719-4d8f-a178-d1d8f0f79788\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega/\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"3d9e40b337b6e30e0b8e81ca60c733b8e0772d11332e013ad9bff337e9a34b5f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"682d3aa8-4e0f-4cb7-9de4-da8a9693ced7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega/\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"92f42c82f66996199cebaa33c98cbc90080b4286809c143987886ff99ad87569\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2300, \"end_char_idx\": 3786, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "doc_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "ref_doc_id": "75e0d8e2-5719-4d8f-a178-d1d8f0f79788", "chroma:document": "## Usage tips\n\n- MEGA can perform quite well with relatively few parameters. See Appendix D in the MEGA paper for examples of architectural specs which perform well in various settings. If using MEGA as a decoder, be sure to set `bidirectional=False` to avoid errors with default bidirectional.\n- Mega-chunk is a variant of mega that reduces time and spaces complexity from quadratic to linear. Utilize chunking with MegaConfig.use_chunking and control chunk size with MegaConfig.chunk_size\n\n\n## Implementation Notes\n\n- The original implementation of MEGA had an inconsistent expectation of attention masks for padding and causal self-attention between the softmax attention and Laplace/squared ReLU method. This implementation addresses that inconsistency.\n- The original implementation did not include token type embeddings; this implementation adds support for these, with the option controlled by MegaConfig.add_token_type_embeddings\n\n\n## MegaConfig\n\n[[autodoc]] MegaConfig\n\n## MegaModel\n\n[[autodoc]] MegaModel\n    - forward\n\n## MegaForCausalLM\n\n[[autodoc]] MegaForCausalLM\n    - forward\n\n## MegaForMaskedLM\n\n[[autodoc]] MegaForMaskedLM\n    - forward\n\n## MegaForSequenceClassification\n\n[[autodoc]] MegaForSequenceClassification\n    - forward\n\n## MegaForMultipleChoice\n\n[[autodoc]] MegaForMultipleChoice\n    - forward\n\n## MegaForTokenClassification\n\n[[autodoc]] MegaForTokenClassification\n    - forward\n\n## MegaForQuestionAnswering\n\n[[autodoc]] MegaForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8e69addb-b6ce-4b2f-8b51-3cbf22be844a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8e69addb-b6ce-4b2f-8b51-3cbf22be844a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen/\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c56bb5fa-adc6-4292-b4aa-b961feee0470\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen/\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f5960213f2e4ed96b6a26343eac693534b3f6e36ce2557d7b3d07c8e88dc0f2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd835cff-3ee7-4bcd-9526-a0b9866cfaab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6e6b44f9559abe8059e29715d25697fef0aa9ce310414362715890a42d84629\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2841, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "doc_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "ref_doc_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "chroma:document": "# CodeGen\n\n## Overview\n\nThe CodeGen model was proposed in [A Conversational Paradigm for Program Synthesis](https://arxiv.org/abs/2203.13474) by Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.\n\nCodeGen is an autoregressive language model for program synthesis trained sequentially on [The Pile](https://pile.eleuther.ai/), BigQuery, and BigPython.\n\nThe abstract from the paper is the following:\n\n*Program synthesis strives to generate a computer program as a solution to a given problem specification. We propose a conversational program synthesis approach via large language models, which addresses the challenges of searching over a vast program space and user intent specification faced in prior approaches. Our new approach casts the process of writing a specification and program as a multi-turn conversation between a user and a system. It treats program synthesis as a sequence prediction problem, in which the specification is expressed in natural language and the desired program is conditionally sampled. We train a family of large language models, called CodeGen, on natural language and programming language data. With weak supervision in the data and the scaling up of data size and model size, conversational capacities emerge from the simple autoregressive language modeling. To study the model behavior on conversational program synthesis, we develop a multi-turn programming benchmark (MTPB), where solving each problem requires multi-step synthesis via multi-turn conversation between the user and the model. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm. In addition, our model CodeGen (with up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the HumanEval benchmark. We make the training library JaxFormer including checkpoints available as open source contribution: [this https URL](https://github.com/salesforce/codegen).* \n\nThis model was contributed by [Hiroaki Hayashi](https://huggingface.co/rooa).\nThe original code can be found [here](https://github.com/salesforce/codegen).\n\n## Checkpoint Naming\n\n* CodeGen model [checkpoints](https://huggingface.co/models?other=codegen) are available on different pre-training data with variable sizes.\n* The format is: `Salesforce/codegen-{size}-{data}`, where\n  * `size`: `350M`, `2B`, `6B`, `16B`\n  * `data`: \n    * `nl`: Pre-trained on the Pile\n    * `multi`: Initialized with `nl`, then further pre-trained on multiple programming languages data\n    * `mono`: Initialized with `multi`, then further pre-trained on Python data\n* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: cd835cff-3ee7-4bcd-9526-a0b9866cfaab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/codegen/", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"cd835cff-3ee7-4bcd-9526-a0b9866cfaab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen/\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c56bb5fa-adc6-4292-b4aa-b961feee0470\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen/\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"8f5960213f2e4ed96b6a26343eac693534b3f6e36ce2557d7b3d07c8e88dc0f2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e69addb-b6ce-4b2f-8b51-3cbf22be844a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen/\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"67b6a0662f7aa5517fa47a59ea6fb98bd4dde385fbf25541d32b833e1f8b11b9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1076, \"end_char_idx\": 3755, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "doc_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "ref_doc_id": "c56bb5fa-adc6-4292-b4aa-b961feee0470", "chroma:document": "We train a family of large language models, called CodeGen, on natural language and programming language data. With weak supervision in the data and the scaling up of data size and model size, conversational capacities emerge from the simple autoregressive language modeling. To study the model behavior on conversational program synthesis, we develop a multi-turn programming benchmark (MTPB), where solving each problem requires multi-step synthesis via multi-turn conversation between the user and the model. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm. In addition, our model CodeGen (with up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the HumanEval benchmark. We make the training library JaxFormer including checkpoints available as open source contribution: [this https URL](https://github.com/salesforce/codegen).* \n\nThis model was contributed by [Hiroaki Hayashi](https://huggingface.co/rooa).\nThe original code can be found [here](https://github.com/salesforce/codegen).\n\n## Checkpoint Naming\n\n* CodeGen model [checkpoints](https://huggingface.co/models?other=codegen) are available on different pre-training data with variable sizes.\n* The format is: `Salesforce/codegen-{size}-{data}`, where\n  * `size`: `350M`, `2B`, `6B`, `16B`\n  * `data`: \n    * `nl`: Pre-trained on the Pile\n    * `multi`: Initialized with `nl`, then further pre-trained on multiple programming languages data\n    * `mono`: Initialized with `multi`, then further pre-trained on Python data\n* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python.\n\n## Usage example\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> checkpoint = \"Salesforce/codegen-350M-mono\"\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n\n>>> text = \"def hello_world():\"\n\n>>> completion = model.generate(**tokenizer(text, return_tensors=\"pt\"))\n\n>>> print(tokenizer.decode(completion[0]))\ndef hello_world():\n    print(\"Hello World\")\n\nhello_world()\n```\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## CodeGenConfig\n\n[[autodoc]] CodeGenConfig\n    - all\n\n## CodeGenTokenizer\n\n[[autodoc]] CodeGenTokenizer\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CodeGenTokenizerFast\n\n[[autodoc]] CodeGenTokenizerFast\n\n## CodeGenModel\n\n[[autodoc]] CodeGenModel\n    - forward\n\n## CodeGenForCausalLM\n\n[[autodoc]] CodeGenForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 4451d02c-d07a-422e-ae2d-c92c50ee87a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"4451d02c-d07a-422e-ae2d-c92c50ee87a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb400873-3e05-4025-8800-fcde494f31de\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"06a49eae299cbe977a8db5498e8706b78b041dfa1ca26991b1aa6c6b1be911e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2844, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb400873-3e05-4025-8800-fcde494f31de", "doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "ref_doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "chroma:document": "# Cohere\n\n## Overview\n\nThe Cohere Command-R model was proposed in the blogpost [Command-R: Retrieval Augmented Generation at Production Scale](https://txt.cohere.com/command-r/) by the Cohere Team.\n\nThe abstract from the paper is the following:\n\n*Command-R is a scalable generative model targeting RAG and Tool Use to enable production-scale AI for enterprise. Today, we are introducing Command-R, a new LLM aimed at large-scale production workloads. Command-R targets the emerging \u201cscalable\u201d category of models that balance high efficiency with strong accuracy, enabling companies to move beyond proof of concept, and into production.*\n\n*Command-R is a generative model optimized for long context tasks such as retrieval augmented generation (RAG) and using external APIs and tools. It is designed to work in concert with our industry-leading Embed and Rerank models to provide best-in-class integration for RAG applications and excel at enterprise use cases. As a model built for companies to implement at scale, Command-R boasts:\n- Strong accuracy on RAG and Tool Use\n- Low latency, and high throughput\n- Longer 128k context and lower pricing\n- Strong capabilities across 10 key languages\n- Model weights available on HuggingFace for research and evaluation\n\nCheckout model checkpoints [here](https://huggingface.co/CohereForAI/c4ai-command-r-v01).\nThis model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet \u00dcst\u00fcn](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\nThe model and tokenizer can be loaded via:\n\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb400873-3e05-4025-8800-fcde494f31de\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4451d02c-d07a-422e-ae2d-c92c50ee87a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"56b743109eeacbf2ea86d7fce5b32291e2c53ce7f5ef0d2cd785b28cbf5095eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c0bcbbe2-ecf0-4c56-9844-344657e6a0df\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98896c33887a60da0521ff2a10d42500790e0aa73423af8aeeb260cdaef48621\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1352, \"end_char_idx\": 4038, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb400873-3e05-4025-8800-fcde494f31de", "doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "ref_doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "chroma:document": "This model was contributed by [Saurabh Dash](https://huggingface.co/saurabhdash) and [Ahmet \u00dcst\u00fcn](https://huggingface.co/ahmetustun). The code of the implementation in Hugging Face is based on GPT-NeoX [here](https://github.com/EleutherAI/gpt-neox).\n\n## Usage tips\n\n<Tip warning={true}>\n\nThe checkpoints uploaded on the Hub use `torch_dtype = 'float16'`, which will be\nused by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. \n\nThe `dtype` of the online weights is mostly irrelevant unless you are using `torch_dtype=\"auto\"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained(\"path\", torch_dtype = \"auto\")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online), then it will be casted to the default `dtype` of `torch` (becomes `torch.float32`), and finally, if there is a `torch_dtype` provided in the config, it will be used. \n\nTraining the model in `float16` is not recommended and is known to produce `nan`; as such, the model should be trained in `bfloat16`.\n\n</Tip>\nThe model and tokenizer can be loaded via:\n\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: c0bcbbe2-ecf0-4c56-9844-344657e6a0df
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"c0bcbbe2-ecf0-4c56-9844-344657e6a0df\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb400873-3e05-4025-8800-fcde494f31de\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f115da5-60e5-4ec3-9ca8-19fbfd9f4efc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"15e43647aca6fbfc69b3b819263f5d911defb4db31d1cf56e7d2e0fbcada12a9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92e559ab-f011-42ee-a39a-352ddf20e8c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"15e6216aa2d9b487337d0d692fd8cf2bad2093eadcd639f533de92c2e2180395\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2429, \"end_char_idx\": 5500, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb400873-3e05-4025-8800-fcde494f31de", "doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "ref_doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "chroma:document": "</Tip>\nThe model and tokenizer can be loaded via:\n\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Command-R. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n\n<PipelineTag pipeline=\"text-generation\"/>\n\nLoading FP16 model\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\nLoading bitsnbytes 4bit quantized model\n```python\n# pip install transformers bitsandbytes accelerate\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nbnb_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 92e559ab-f011-42ee-a39a-352ddf20e8c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/cohere/", "title": "Cohere", "tokens": 1441, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"92e559ab-f011-42ee-a39a-352ddf20e8c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb400873-3e05-4025-8800-fcde494f31de\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6de0fa4944977727119bf793f0eda3717fb96331b80ea3752a9c62d6e387d991\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c0bcbbe2-ecf0-4c56-9844-344657e6a0df\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/cohere/\", \"title\": \"Cohere\", \"tokens\": 1441, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"74dd728b49c02d8afa11a7807a6b9a69a9cf8d52600a6cdf83b4481e76b82940\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4041, \"end_char_idx\": 5886, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb400873-3e05-4025-8800-fcde494f31de", "doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "ref_doc_id": "eb400873-3e05-4025-8800-fcde494f31de", "chroma:document": "<PipelineTag pipeline=\"text-generation\"/>\n\nLoading FP16 model\n```python\n# pip install transformers\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n\n# Format message with the command-r chat template\nmessages = [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}]\ninput_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n## <BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>Hello, how are you?<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\nLoading bitsnbytes 4bit quantized model\n```python\n# pip install transformers bitsandbytes accelerate\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nbnb_config = BitsAndBytesConfig(load_in_4bit=True)\n\nmodel_id = \"CohereForAI/c4ai-command-r-v01\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)\n\ngen_tokens = model.generate(\n    input_ids, \n    max_new_tokens=100, \n    do_sample=True, \n    temperature=0.3,\n    )\n\ngen_text = tokenizer.decode(gen_tokens[0])\nprint(gen_text)\n```\n\n\n## CohereConfig\n\n[[autodoc]] CohereConfig\n\n## CohereTokenizerFast\n\n[[autodoc]] CohereTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## CohereModel\n\n[[autodoc]] CohereModel\n    - forward\n\n\n## CohereForCausalLM\n\n[[autodoc]] CohereForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2ad42a43-3169-4abc-a60d-fb6ac3118f56
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2ad42a43-3169-4abc-a60d-fb6ac3118f56\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet/\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c6a2ce7-fe40-4d40-84be-3389c8d03fd2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet/\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d428f6f790b0762383386442074bc8ee4074011b4d0dfe60e8dea6e67cfbd4f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6d5e0a73-a831-4fc7-9e5d-291e7d98f67c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3d9af46590fc64f156313f524aa4e75ad480d6763b22a922cfd7b10ba3e2589\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2353, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "doc_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "ref_doc_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "chroma:document": "# ProphetNet\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=prophetnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-prophetnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/prophetnet-large-uncased\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ProphetNet model was proposed in [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei\nZhang, Ming Zhou on 13 Jan, 2020.\n\nProphetNet is an encoder-decoder model and can predict n-future tokens for \"ngram\" language modeling instead of just\nthe next token.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel\nself-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of\nthe optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by\nn-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time\nstep. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent\noverfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale\ndataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for\nabstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new\nstate-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*\n\nThe Authors' code can be found [here](https://github.com/microsoft/ProphetNet).\n\n## Usage tips\n\n- ProphetNet is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- The model architecture is based on the original Transformer, but replaces the \u201cstandard\u201d self-attention mechanism in the decoder by a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 6d5e0a73-a831-4fc7-9e5d-291e7d98f67c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/prophetnet/", "title": "ProphetNet", "tokens": 993, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"6d5e0a73-a831-4fc7-9e5d-291e7d98f67c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet/\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c6a2ce7-fe40-4d40-84be-3389c8d03fd2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet/\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d428f6f790b0762383386442074bc8ee4074011b4d0dfe60e8dea6e67cfbd4f9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ad42a43-3169-4abc-a60d-fb6ac3118f56\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/prophetnet/\", \"title\": \"ProphetNet\", \"tokens\": 993, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d3c38a7b35cea025c6567465e85fa303b023c640c6676022bedd7f272723f655\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 673, \"end_char_idx\": 3365, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "doc_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "ref_doc_id": "0c6a2ce7-fe40-4d40-84be-3389c8d03fd2", "chroma:document": "ProphetNet is an encoder-decoder model and can predict n-future tokens for \"ngram\" language modeling instead of just\nthe next token.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel\nself-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of\nthe optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by\nn-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time\nstep. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent\noverfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale\ndataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for\nabstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new\nstate-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*\n\nThe Authors' code can be found [here](https://github.com/microsoft/ProphetNet).\n\n## Usage tips\n\n- ProphetNet is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than\n  the left.\n- The model architecture is based on the original Transformer, but replaces the \u201cstandard\u201d self-attention mechanism in the decoder by a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism.\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## ProphetNetConfig\n\n[[autodoc]] ProphetNetConfig\n\n## ProphetNetTokenizer\n\n[[autodoc]] ProphetNetTokenizer\n\n## ProphetNet specific outputs\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqLMOutput\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetSeq2SeqModelOutput\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetDecoderModelOutput\n\n[[autodoc]] models.prophetnet.modeling_prophetnet.ProphetNetDecoderLMOutput\n\n## ProphetNetModel\n\n[[autodoc]] ProphetNetModel\n    - forward\n\n## ProphetNetEncoder\n\n[[autodoc]] ProphetNetEncoder\n    - forward\n\n## ProphetNetDecoder\n\n[[autodoc]] ProphetNetDecoder\n    - forward\n\n## ProphetNetForConditionalGeneration\n\n[[autodoc]] ProphetNetForConditionalGeneration\n    - forward\n\n## ProphetNetForCausalLM\n\n[[autodoc]] ProphetNetForCausalLM\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a8557bde-3dec-4d16-ae21-d1a0c9ad887b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a8557bde-3dec-4d16-ae21-d1a0c9ad887b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8a19e8fb-f4a9-4e18-a39e-18044058e498\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ea3143e15481c06ef3dbe0ea85abe948696006a18018d72b934560e364f4f3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2773, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "# GPT-NeoX\n\n## Overview\n\nWe introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will\nbe made freely and openly available to the public through a permissive license. It is, to the best of our knowledge,\nthe largest dense autoregressive model that has publicly available weights at the time of submission. In this work,\nwe describe GPT-NeoX-20B's architecture and training and evaluate its performance on a range of language-understanding,\nmathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and\ngains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source\nthe training and evaluation code, as well as the model weights, at [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).\n\nDevelopment of the model was led by Sid Black, Stella Biderman and Eric Hallahan, and the model was trained with\ngenerous the support of [CoreWeave](https://www.coreweave.com/).\n\nGPT-NeoX-20B was trained with fp16, thus it is recommended to initialize the model as follows:\n\n```python\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\").half().cuda()\n```\n\nGPT-NeoX-20B also has a different tokenizer from the one used in GPT-J-6B and GPT-Neo. The new tokenizer allocates\nadditional tokens to whitespace characters, making the model more suitable for certain tasks like code generation.\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT Neo model.\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\n>>> model = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n>>> tokenizer = GPTNeoXTokenizerFast.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n\n>>> prompt = \"GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 8a19e8fb-f4a9-4e18-a39e-18044058e498
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"8a19e8fb-f4a9-4e18-a39e-18044058e498\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a8557bde-3dec-4d16-ae21-d1a0c9ad887b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5a84f17abcf55e69a23b745e197860cfad8a84f99f1021a35aeb8c97adb9f086\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"583f6358-24ee-4f5a-83b8-a92e44747723\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"66d0a5e20132381cda993924da98df5706c9cfe258c1253a834c9698cb749e98\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1341, \"end_char_idx\": 4350, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "The new tokenizer allocates\nadditional tokens to whitespace characters, making the model more suitable for certain tasks like code generation.\n\n## Usage example\n\nThe `generate()` method can be used to generate text using GPT Neo model.\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\n>>> model = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n>>> tokenizer = GPTNeoXTokenizerFast.from_pretrained(\"EleutherAI/gpt-neox-20b\")\n\n>>> prompt = \"GPTNeoX20B is a 20B-parameter autoregressive Transformer model developed by EleutherAI.\"\n\n>>> input_ids = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\n>>> gen_tokens = model.generate(\n...     input_ids,\n...     do_sample=True,\n...     temperature=0.9,\n...     max_length=100,\n... )\n>>> gen_text = tokenizer.batch_decode(gen_tokens)[0]\n```\n\n## Using Flash Attention 2\n\nFlash Attention 2 is an faster, optimized version of the model.\n\n### Installation \n\nFirst, check whether your hardware is compatible with Flash Attention 2. The latest list of compatible hardware can be found in the [official documentation](https://github.com/Dao-AILab/flash-attention#installation-and-features). If your hardware is not compatible with Flash Attention 2, you can still benefit from attention kernel optimisations through Better Transformer support covered [above](https://huggingface.co/docs/transformers/main/en/model_doc/bark#using-better-transformer).\n\nNext, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `stockmark/gpt-neox-japanese-1.4b` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/gpt-neox-1.8b-speedup.jpg\">\n</div>\n\n\n## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 583f6358-24ee-4f5a-83b8-a92e44747723
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"583f6358-24ee-4f5a-83b8-a92e44747723\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8a19e8fb-f4a9-4e18-a39e-18044058e498\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"35249976ad59ca00ddf95e2fc4a411440742d598ff69ad9d83e1862797a7d6dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5e8c0589-e474-4444-8550-fe374ce7e9db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aed132cddf7d762190785d510ade4241d0f314cf5e4b6edfeac5f95304ace185\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2775, \"end_char_idx\": 5792, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "Next, [install](https://github.com/Dao-AILab/flash-attention#installation-and-features) the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\n### Usage\n\nTo load a model using Flash Attention 2, we can pass the argument `attn_implementation=\"flash_attention_2\"` to [`.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained). We'll also load the model in half-precision (e.g. `torch.float16`), since it results in almost no degradation to audio quality but significantly lower memory usage and faster inference:\n\n```python\n>>> from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(device)\n...\n```\n\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `stockmark/gpt-neox-japanese-1.4b` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/gpt-neox-1.8b-speedup.jpg\">\n</div>\n\n\n## Using Scaled Dot Product Attention (SDPA)\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function\nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import GPTNeoXForCausalLM\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped), we saw the\nfollowing speedups during training and inference.\n\n### Training\n| Batch size |    Seq len | Time per batch (Eager - s) |    Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |\n|-----------:|-----------:|---------------------------:|-----------------------------:|------------:|--------------------:|-------------------:|------------------:|\n|          1 |        128 |                      0.024 |                        0."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5e8c0589-e474-4444-8550-fe374ce7e9db
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5e8c0589-e474-4444-8550-fe374ce7e9db\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"583f6358-24ee-4f5a-83b8-a92e44747723\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"06b4a2dbc99446cb67e87c257cdecb853b25c83a9e08d76a72d818e2c5a07f22\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"08d249c5-84b6-4758-8092-d86c16a2e54e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d5aa62e8f963b5cf99315654a67f2a32fef652371576c01c0b9b70b0644cdd45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4351, \"end_char_idx\": 7126, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "See the\n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html)\nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set\n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```python\nfrom transformers import GPTNeoXForCausalLM\nmodel = GPTNeoXForCausalLM.from_pretrained(\"EleutherAI/gpt-neox-20b\", torch_dtype=torch.float16, attn_implementation=\"sdpa\")\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (rtx3080ti-16GB, PyTorch 2.2.1, OS Ubuntu 22.04) using `float16` with\n[pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped), we saw the\nfollowing speedups during training and inference.\n\n### Training\n| Batch size |    Seq len | Time per batch (Eager - s) |    Time per batch (SDPA - s) | Speedup (%) | Eager peak mem (MB) | SDPA peak mem (MB) |    Mem saving (%) |\n|-----------:|-----------:|---------------------------:|-----------------------------:|------------:|--------------------:|-------------------:|------------------:|\n|          1 |        128 |                      0.024 |                        0.019 |      28.945 |             1789.95 |            1789.95 |                 0 |\n|          1 |        256 |                      0.039 |                        0.031 |       23.18 |             1845.83 |            1844.84 |             0.053 |\n|          1 |        512 |                       0.08 |                        0.055 |      45.524 |             2278.38 |            1953.76 |            16.615 |\n|          1 |       1024 |                       0.19 |                        0.102 |      86.777 |             4772.36 |            2408.35 |            98.159 |\n|          1 |       2048 |                      0.565 |                        0.204 |     177.098 |             13484.1 |            3882.01 |           247.348 |\n|          2 |        128 |                      0.037 |                        0.032 |      15.121 |             1843.86 |            1844.78 |             -0.05 |\n|          2 |        256 |                      0.067 |                        0.055 |      21.706 |             1999.72 |            1951.67 |             2.462 |\n|          2 |        512 |                      0.144 |                        0.096 |      50.046 |             3613.16 |            2406.77 |            50.125 |\n|          2 |       1024 |                      0.366 |                        0.193 |      89."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 08d249c5-84b6-4758-8092-d86c16a2e54e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"08d249c5-84b6-4758-8092-d86c16a2e54e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5e8c0589-e474-4444-8550-fe374ce7e9db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f8bce7e3b86e723d7b2054b806ec6ef40b77144ba092d1b7c3c2be8d25d9e57d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a23ac3b4-f83b-45ae-9ffc-8569806b4a45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4a7402e1aa1033db1de0865a8b3e08b68165389a06bd26544496bc1eead35b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5761, \"end_char_idx\": 8642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "024 |                        0.019 |      28.945 |             1789.95 |            1789.95 |                 0 |\n|          1 |        256 |                      0.039 |                        0.031 |       23.18 |             1845.83 |            1844.84 |             0.053 |\n|          1 |        512 |                       0.08 |                        0.055 |      45.524 |             2278.38 |            1953.76 |            16.615 |\n|          1 |       1024 |                       0.19 |                        0.102 |      86.777 |             4772.36 |            2408.35 |            98.159 |\n|          1 |       2048 |                      0.565 |                        0.204 |     177.098 |             13484.1 |            3882.01 |           247.348 |\n|          2 |        128 |                      0.037 |                        0.032 |      15.121 |             1843.86 |            1844.78 |             -0.05 |\n|          2 |        256 |                      0.067 |                        0.055 |      21.706 |             1999.72 |            1951.67 |             2.462 |\n|          2 |        512 |                      0.144 |                        0.096 |      50.046 |             3613.16 |            2406.77 |            50.125 |\n|          2 |       1024 |                      0.366 |                        0.193 |      89.666 |             8707.55 |            3878.86 |           124.487 |\n|          2 |       2048 |                        OOM |                        0.379 |           / |                 OOM |            6825.13 | SDPA does not OOM |\n|          4 |        128 |                       0.06 |                        0.054 |      11.539 |              1947.6 |            1952.06 |            -0.228 |\n|          4 |        256 |                      0.119 |                        0.093 |      28.072 |             3008.39 |            2405.99 |            25.038 |\n|          4 |        512 |                      0.275 |                        0.187 |      47.145 |             6290.58 |            3877.29 |            62.242 |\n|          4 |       1024 |                        OOM |                         0.36 |           / |                 OOM |            6821.98 | SDPA does not OOM |\n|          4 |       2048 |                        OOM |                        0.731 |           / |                 OOM |            12705.1 | SDPA does not OOM |\n\n### Inference\n|    Batch size |      Seq len |    Per token latency Eager (ms) |    Per token latency SDPA (ms) |    Speedup (%) |    Mem Eager (MB) |   Mem SDPA (MB) |    Mem saved (%) |\n|--------------:|-------------:|--------------------------------:|-------------------------------:|---------------:|------------------:|----------------:|-----------------:|\n|             1 |          128 |                           6.569 |                          5."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a23ac3b4-f83b-45ae-9ffc-8569806b4a45
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a23ac3b4-f83b-45ae-9ffc-8569806b4a45\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"08d249c5-84b6-4758-8092-d86c16a2e54e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"737f88890d00fbf5d457c75fe4f3ada7d9647d73c3159fa3f6dd3e4e77fdceab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b996b549-3348-4c3e-89f1-751497e56450\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25c80eb907589a49695a45bc851b5bb9e2419288568874be07af4329f1542df3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7112, \"end_char_idx\": 10051, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "193 |      89.666 |             8707.55 |            3878.86 |           124.487 |\n|          2 |       2048 |                        OOM |                        0.379 |           / |                 OOM |            6825.13 | SDPA does not OOM |\n|          4 |        128 |                       0.06 |                        0.054 |      11.539 |              1947.6 |            1952.06 |            -0.228 |\n|          4 |        256 |                      0.119 |                        0.093 |      28.072 |             3008.39 |            2405.99 |            25.038 |\n|          4 |        512 |                      0.275 |                        0.187 |      47.145 |             6290.58 |            3877.29 |            62.242 |\n|          4 |       1024 |                        OOM |                         0.36 |           / |                 OOM |            6821.98 | SDPA does not OOM |\n|          4 |       2048 |                        OOM |                        0.731 |           / |                 OOM |            12705.1 | SDPA does not OOM |\n\n### Inference\n|    Batch size |      Seq len |    Per token latency Eager (ms) |    Per token latency SDPA (ms) |    Speedup (%) |    Mem Eager (MB) |   Mem SDPA (MB) |    Mem saved (%) |\n|--------------:|-------------:|--------------------------------:|-------------------------------:|---------------:|------------------:|----------------:|-----------------:|\n|             1 |          128 |                           6.569 |                          5.858 |          12.14 |           974.831 |         974.826 |                0 |\n|             1 |          256 |                           7.009 |                          5.863 |         19.542 |           1029.01 |         1028.08 |             0.09 |\n|             1 |          512 |                           7.157 |                          5.965 |         19.983 |           1137.54 |         1137.52 |            0.001 |\n|             1 |         1024 |                           7.523 |                          6.506 |         15.637 |            1329.3 |         1329.26 |            0.003 |\n|             1 |         2048 |                           9.271 |                          9.205 |          0.713 |           1752.47 |         1734.51 |            1.036 |\n|             2 |          128 |                           7.239 |                          5.959 |         21.493 |            1044.8 |         1028.37 |            1.597 |\n|             2 |          256 |                           7.228 |                          6.036 |         19.757 |           1167.32 |         1137.73 |            2.601 |\n|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |\n|             2 |         1024 |                           8.916 |                          8.632 |          3."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b996b549-3348-4c3e-89f1-751497e56450
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b996b549-3348-4c3e-89f1-751497e56450\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a23ac3b4-f83b-45ae-9ffc-8569806b4a45\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"717ab8f06a836e7b1f96ef1060819369bac0c8b643c9b0574c628fb274897a66\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"27d68e86-e731-4558-9451-916c04fe8949\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"70c6c0b621f7905838cfa95f82bb2e260928033966bce8fe6c72fcfac3e9a074\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8609, \"end_char_idx\": 11157, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "569 |                          5.858 |          12.14 |           974.831 |         974.826 |                0 |\n|             1 |          256 |                           7.009 |                          5.863 |         19.542 |           1029.01 |         1028.08 |             0.09 |\n|             1 |          512 |                           7.157 |                          5.965 |         19.983 |           1137.54 |         1137.52 |            0.001 |\n|             1 |         1024 |                           7.523 |                          6.506 |         15.637 |            1329.3 |         1329.26 |            0.003 |\n|             1 |         2048 |                           9.271 |                          9.205 |          0.713 |           1752.47 |         1734.51 |            1.036 |\n|             2 |          128 |                           7.239 |                          5.959 |         21.493 |            1044.8 |         1028.37 |            1.597 |\n|             2 |          256 |                           7.228 |                          6.036 |         19.757 |           1167.32 |         1137.73 |            2.601 |\n|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |\n|             2 |         1024 |                           8.916 |                          8.632 |          3.291 |           1752.56 |         1734.62 |            1.034 |\n|             2 |         2048 |                          12.628 |                         12.606 |          0.181 |           2558.72 |          2545.8 |            0.508 |\n|             4 |          128 |                           7.278 |                          6.046 |         20.373 |           1168.41 |         1137.79 |            2.691 |\n|             4 |          256 |                           7.614 |                          6.588 |         15.574 |            1353.1 |         1329.79 |            1.753 |\n|             4 |          512 |                           8.798 |                          8.144 |          8.028 |           1752.76 |         1734.85 |            1.032 |\n|             4 |         1024 |                          11.765 |                         11.303 |           4.09 |           2558.96 |         2546.04 |            0.508 |\n|             4 |         2048 |                          19.568 |                         17.735 |          10.33 |            4175.5 |         4165.26 |            0.246 |"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 27d68e86-e731-4558-9451-916c04fe8949
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_neox/", "title": "GPT-NeoX", "tokens": 3264, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"27d68e86-e731-4558-9451-916c04fe8949\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"64e03c88-b399-49f2-9774-a8a26b4a2b60\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"19b9f8a6bf264cadb08aa8f34388a8b3ffd7278fc158b4916cb42d25f56ca3aa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b996b549-3348-4c3e-89f1-751497e56450\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_neox/\", \"title\": \"GPT-NeoX\", \"tokens\": 3264, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"f11d9cb6fdf08041c781fe8a188651e75124fbd1e67c7e64f2e692efad247639\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9724, \"end_char_idx\": 11746, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "ref_doc_id": "64e03c88-b399-49f2-9774-a8a26b4a2b60", "chroma:document": "32 |         1137.73 |            2.601 |\n|             2 |          512 |                           7.538 |                          6.693 |         12.628 |           1352.93 |         1329.55 |            1.758 |\n|             2 |         1024 |                           8.916 |                          8.632 |          3.291 |           1752.56 |         1734.62 |            1.034 |\n|             2 |         2048 |                          12.628 |                         12.606 |          0.181 |           2558.72 |          2545.8 |            0.508 |\n|             4 |          128 |                           7.278 |                          6.046 |         20.373 |           1168.41 |         1137.79 |            2.691 |\n|             4 |          256 |                           7.614 |                          6.588 |         15.574 |            1353.1 |         1329.79 |            1.753 |\n|             4 |          512 |                           8.798 |                          8.144 |          8.028 |           1752.76 |         1734.85 |            1.032 |\n|             4 |         1024 |                          11.765 |                         11.303 |           4.09 |           2558.96 |         2546.04 |            0.508 |\n|             4 |         2048 |                          19.568 |                         17.735 |          10.33 |            4175.5 |         4165.26 |            0.246 |\n\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## GPTNeoXConfig\n\n[[autodoc]] GPTNeoXConfig\n\n## GPTNeoXTokenizerFast\n\n[[autodoc]] GPTNeoXTokenizerFast\n\n## GPTNeoXModel\n\n[[autodoc]] GPTNeoXModel\n    - forward\n\n## GPTNeoXForCausalLM\n\n[[autodoc]] GPTNeoXForCausalLM\n    - forward\n\n## GPTNeoXForQuestionAnswering\n\n[[autodoc]] GPTNeoXForQuestionAnswering\n    - forward\n\n## GPTNeoXForSequenceClassification\n\n[[autodoc]] GPTNeoXForSequenceClassification\n    - forward\n\n## GPTNeoXForTokenClassification\n\n[[autodoc]] GPTNeoXForTokenClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: adeb55c5-134c-423f-8ea6-61d25381be36
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/fsmt/", "title": "FSMT", "tokens": 619, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"adeb55c5-134c-423f-8ea6-61d25381be36\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fsmt/\", \"title\": \"FSMT\", \"tokens\": 619, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e7e2bc-6e51-473b-ae86-5076a6bccdda\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/fsmt/\", \"title\": \"FSMT\", \"tokens\": 619, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9c1dd6154a2721cf765d47aef416534cb2446ee17b54fed9ff9f9fd0b3569356\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1964, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e7e2bc-6e51-473b-ae86-5076a6bccdda", "doc_id": "b0e7e2bc-6e51-473b-ae86-5076a6bccdda", "ref_doc_id": "b0e7e2bc-6e51-473b-ae86-5076a6bccdda", "chroma:document": "# FSMT\n\n## Overview\n\nFSMT (FairSeq MachineTranslation) models were introduced in [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616) by Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, Sergey Edunov.\n\nThe abstract of the paper is the following:\n\n*This paper describes Facebook FAIR's submission to the WMT19 shared news translation task. We participate in two\nlanguage pairs and four language directions, English <-> German and English <-> Russian. Following our submission from\nlast year, our baseline systems are large BPE-based transformer models trained with the Fairseq sequence modeling\ntoolkit which rely on sampled back-translations. This year we experiment with different bitext data filtering schemes,\nas well as with adding filtered back-translated data. We also ensemble and fine-tune our models on domain-specific\ndata, then decode using noisy channel model reranking. Our submissions are ranked first in all four directions of the\nhuman evaluation campaign. On En->De, our system significantly outperforms other systems as well as human translations.\nThis system improves upon our WMT'18 submission by 4.5 BLEU points.*\n\nThis model was contributed by [stas](https://huggingface.co/stas). The original code can be found\n[here](https://github.com/pytorch/fairseq/tree/master/examples/wmt19).\n\n## Implementation Notes\n\n- FSMT uses source and target vocabulary pairs that aren't combined into one. It doesn't share embeddings tokens\n  either. Its tokenizer is very similar to [`XLMTokenizer`] and the main model is derived from\n  [`BartModel`].\n\n\n## FSMTConfig\n\n[[autodoc]] FSMTConfig\n\n## FSMTTokenizer\n\n[[autodoc]] FSMTTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## FSMTModel\n\n[[autodoc]] FSMTModel\n    - forward\n\n## FSMTForConditionalGeneration\n\n[[autodoc]] FSMTForConditionalGeneration\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2ccc31b0-7d91-4041-b4c4-a753b4a0227e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2ccc31b0-7d91-4041-b4c4-a753b4a0227e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5/\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c8379d1-0f28-4fad-a8bd-e06bcefaa776\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5/\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ab0e68bf60af7098408bedf6b9e850e0d1f33e9dfc6422e4c145f247b70b0f0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f5cbaf6-8edd-4364-a9ca-864de9a279c4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"40cdcd1a879c8d17fb25945fd2d6ea4018dd8551dbf02b230be59f85def6497e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2635, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "doc_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "ref_doc_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "chroma:document": "# mT5\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=mt5\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-mt5-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/mt5-small-finetuned-arxiv-cs-finetuned-arxiv-cs-full\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe mT5 model was presented in [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934) by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya\nSiddhant, Aditya Barua, Colin Raffel.\n\nThe abstract from the paper is the following:\n\n*The recent \"Text-to-Text Transfer Transformer\" (T5) leveraged a unified text-to-text format and scale to attain\nstate-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a\nmultilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail\nthe design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual\nbenchmarks. We also describe a simple technique to prevent \"accidental translation\" in the zero-shot setting, where a\ngenerative model chooses to (partially) translate its prediction into the wrong language. All of the code and model\ncheckpoints used in this work are publicly available.*\n\nNote: mT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.\nTherefore, this model has to be fine-tuned before it is usable on a downstream task, unlike the original T5 model.\nSince mT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task\nfine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.\n\nGoogle has released the following variants:\n\n- [google/mt5-small](https://huggingface.co/google/mt5-small)\n\n- [google/mt5-base](https://huggingface.co/google/mt5-base)\n\n- [google/mt5-large](https://huggingface.co/google/mt5-large)\n\n- [google/mt5-xl](https://huggingface.co/google/mt5-xl)\n\n- [google/mt5-xxl](https://huggingface.co/google/mt5-xxl).\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be\nfound [here](https://github.com/google-research/multilingual-t5).\n\n## Resources\n\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## MT5Config\n\n[[autodoc]] MT5Config\n\n## MT5Tokenizer\n\n[[autodoc]] MT5Tokenizer\n\nSee [`T5Tokenizer`] for all details."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 5f5cbaf6-8edd-4364-a9ca-864de9a279c4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mt5/", "title": "mT5", "tokens": 1157, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"5f5cbaf6-8edd-4364-a9ca-864de9a279c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5/\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0c8379d1-0f28-4fad-a8bd-e06bcefaa776\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5/\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"5ab0e68bf60af7098408bedf6b9e850e0d1f33e9dfc6422e4c145f247b70b0f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ccc31b0-7d91-4041-b4c4-a753b4a0227e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mt5/\", \"title\": \"mT5\", \"tokens\": 1157, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"a25e6d309e407335169d5bb8b53b13f75b8b7252ecde8b90bd7af39d01ee4330\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2638, \"end_char_idx\": 3523, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "doc_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "ref_doc_id": "0c8379d1-0f28-4fad-a8bd-e06bcefaa776", "chroma:document": "## MT5TokenizerFast\n\n[[autodoc]] MT5TokenizerFast\n\nSee [`T5TokenizerFast`] for all details.\n\n<frameworkcontent>\n<pt>\n\n## MT5Model\n\n[[autodoc]] MT5Model\n\n## MT5ForConditionalGeneration\n\n[[autodoc]] MT5ForConditionalGeneration\n\n## MT5EncoderModel\n\n[[autodoc]] MT5EncoderModel\n\n## MT5ForSequenceClassification\n\n[[autodoc]] MT5ForSequenceClassification\n\n## MT5ForTokenClassification\n\n[[autodoc]] MT5ForTokenClassification\n\n## MT5ForQuestionAnswering\n\n[[autodoc]] MT5ForQuestionAnswering\n\n</pt>\n<tf>\n\n## TFMT5Model\n\n[[autodoc]] TFMT5Model\n\n## TFMT5ForConditionalGeneration\n\n[[autodoc]] TFMT5ForConditionalGeneration\n\n## TFMT5EncoderModel\n\n[[autodoc]] TFMT5EncoderModel\n\n</tf>\n<jax>\n\n## FlaxMT5Model\n\n[[autodoc]] FlaxMT5Model\n\n## FlaxMT5ForConditionalGeneration\n\n[[autodoc]] FlaxMT5ForConditionalGeneration\n\n## FlaxMT5EncoderModel\n\n[[autodoc]] FlaxMT5EncoderModel\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 20a709a5-f830-4284-a68c-14c4be27f734
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"20a709a5-f830-4284-a68c-14c4be27f734\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera/\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf77a3e9-d1fc-4c52-a1ca-768798e0af81\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera/\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ad480bf7077c626676bf59ca40b5c1339a4a4ace942c2e647160abf72f8be1c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54818bc1-466f-4554-a6d0-089ac525d335\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"61b5e04bc01bc1c45f0e07a9e1b382e9366f6559a8cf6587865cdd5a40b13285\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "doc_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "ref_doc_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "chroma:document": "# Hiera\n\n## Overview\n\nHiera was proposed in [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://arxiv.org/abs/2306.00989) by Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer\n\nThe paper introduces \"Hiera,\" a hierarchical Vision Transformer that simplifies the architecture of modern hierarchical vision transformers by removing unnecessary components without compromising on accuracy or efficiency. Unlike traditional transformers that add complex vision-specific components to improve supervised classification performance, Hiera demonstrates that such additions, often termed \"bells-and-whistles,\" are not essential for high accuracy. By leveraging a strong visual pretext task (MAE) for pretraining, Hiera retains simplicity and achieves superior accuracy and speed both in inference and training across various image and video recognition tasks. The approach suggests that spatial biases required for vision tasks can be effectively learned through proper pretraining, eliminating the need for added architectural complexity. \n\nThe abstract from the paper is the following:\n\n*Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/hiera_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Hiera architecture. Taken from the <a href=\"https://arxiv.org/abs/2306.00989\">original paper.</a> </small>\n\nThis model was a joint contibution by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [namangarg110](https://huggingface.co/namangarg110). The original code can be found [here] (https://github.com/facebookresearch/hiera).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Hiera. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`HieraForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 54818bc1-466f-4554-a6d0-089ac525d335
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/hiera/", "title": "Hiera", "tokens": 962, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"54818bc1-466f-4554-a6d0-089ac525d335\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera/\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf77a3e9-d1fc-4c52-a1ca-768798e0af81\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera/\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"ad480bf7077c626676bf59ca40b5c1339a4a4ace942c2e647160abf72f8be1c2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"20a709a5-f830-4284-a68c-14c4be27f734\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/hiera/\", \"title\": \"Hiera\", \"tokens\": 962, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"9f7f2d5698ced7d7378179e513844690570841e6062814e55ada48011a27350f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1650, \"end_char_idx\": 3716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "doc_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "ref_doc_id": "bf77a3e9-d1fc-4c52-a1ca-768798e0af81", "chroma:document": "By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/hiera_overview.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Hiera architecture. Taken from the <a href=\"https://arxiv.org/abs/2306.00989\">original paper.</a> </small>\n\nThis model was a joint contibution by [EduardoPacheco](https://huggingface.co/EduardoPacheco) and [namangarg110](https://huggingface.co/namangarg110). The original code can be found [here] (https://github.com/facebookresearch/hiera).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Hiera. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`HieraForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\n## HieraConfig\n\n[[autodoc]] HieraConfig\n\n## HieraModel\n\n[[autodoc]] HieraModel\n    - forward\n\n## HieraForPreTraining\n\n[[autodoc]] HieraForPreTraining\n    - forward\n  \n## HieraForImageClassification\n\n[[autodoc]] HieraForImageClassification\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 2b1925e0-5cb7-4e65-9f96-1a4689e95a80
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"2b1925e0-5cb7-4e65-9f96-1a4689e95a80\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert/\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b94a0cab-0993-4f83-a74a-c5a75493070c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert/\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d04419119925e60feed4a635622f6e561c5d982934b76364e8f7ade284666b36\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a57adba6-fc57-4a49-a739-02b530dab15d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3540d043a01b0486307106a78e2201e6ac349b060dd8d39f363f229f14d35781\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2185, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "doc_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "ref_doc_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "chroma:document": "# ConvBERT\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=convbert\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-convbert-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/conv-bert-base\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nThe ConvBERT model was proposed in [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://arxiv.org/abs/2008.02496) by Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng\nYan.\n\nThe abstract from the paper is the following:\n\n*Pre-trained language models like BERT and its variants have recently achieved impressive performance in various\nnatural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers\nlarge memory footprint and computation cost. Although all its attention heads query on the whole input sequence for\ngenerating the attention map from a global perspective, we observe some heads only need to learn local dependencies,\nwhich means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to\nreplace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the\nrest self-attention heads, form a new mixed attention block that is more efficient at both global and local context\nlearning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that\nConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and\nfewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while\nusing less than 1/4 training cost. Code and pre-trained models will be released.*\n\nThis model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found\nhere: https://github.com/yitu-opensource/ConvBert\n\n## Usage tips\n\nConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert)."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: a57adba6-fc57-4a49-a739-02b530dab15d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convbert/", "title": "ConvBERT", "tokens": 1149, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"a57adba6-fc57-4a49-a739-02b530dab15d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert/\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b94a0cab-0993-4f83-a74a-c5a75493070c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert/\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"d04419119925e60feed4a635622f6e561c5d982934b76364e8f7ade284666b36\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2b1925e0-5cb7-4e65-9f96-1a4689e95a80\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convbert/\", \"title\": \"ConvBERT\", \"tokens\": 1149, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4965c64026a5226bae1338391b9af032eaa0447f0cf5a5f44520a10f88dfce3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 608, \"end_char_idx\": 3884, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "doc_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "ref_doc_id": "b94a0cab-0993-4f83-a74a-c5a75493070c", "chroma:document": "The abstract from the paper is the following:\n\n*Pre-trained language models like BERT and its variants have recently achieved impressive performance in various\nnatural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers\nlarge memory footprint and computation cost. Although all its attention heads query on the whole input sequence for\ngenerating the attention map from a global perspective, we observe some heads only need to learn local dependencies,\nwhich means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to\nreplace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the\nrest self-attention heads, form a new mixed attention block that is more efficient at both global and local context\nlearning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that\nConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and\nfewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while\nusing less than 1/4 training cost. Code and pre-trained models will be released.*\n\nThis model was contributed by [abhishek](https://huggingface.co/abhishek). The original implementation can be found\nhere: https://github.com/yitu-opensource/ConvBert\n\n## Usage tips\n\nConvBERT training tips are similar to those of BERT. For usage tips refer to [BERT documentation](bert).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## ConvBertConfig\n\n[[autodoc]] ConvBertConfig\n\n## ConvBertTokenizer\n\n[[autodoc]] ConvBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## ConvBertTokenizerFast\n\n[[autodoc]] ConvBertTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## ConvBertModel\n\n[[autodoc]] ConvBertModel\n    - forward\n\n## ConvBertForMaskedLM\n\n[[autodoc]] ConvBertForMaskedLM\n    - forward\n\n## ConvBertForSequenceClassification\n\n[[autodoc]] ConvBertForSequenceClassification\n    - forward\n\n## ConvBertForMultipleChoice\n\n[[autodoc]] ConvBertForMultipleChoice\n    - forward\n\n## ConvBertForTokenClassification\n\n[[autodoc]] ConvBertForTokenClassification\n    - forward\n\n## ConvBertForQuestionAnswering\n\n[[autodoc]] ConvBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFConvBertModel\n\n[[autodoc]] TFConvBertModel\n    - call\n\n## TFConvBertForMaskedLM\n\n[[autodoc]] TFConvBertForMaskedLM\n    - call\n\n## TFConvBertForSequenceClassification\n\n[[autodoc]] TFConvBertForSequenceClassification\n    - call\n\n## TFConvBertForMultipleChoice\n\n[[autodoc]] TFConvBertForMultipleChoice\n    - call\n\n## TFConvBertForTokenClassification\n\n[[autodoc]] TFConvBertForTokenClassification\n    - call\n\n## TFConvBertForQuestionAnswering\n\n[[autodoc]] TFConvBertForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: def94f09-5597-45a5-85ef-6aaab359301b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"def94f09-5597-45a5-85ef-6aaab359301b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3835a32e-c2af-46e1-ba3c-6380906746cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e7d81cb44b28241fb97cecd8dd189527f3c1d841c6746057893dfc87956a47fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd3f9099-a39e-4eb1-98be-cf48ffaaf83c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5949f0fc9a6193534d047dde4720e0b6d9ec4a115216cc5a020a4b8f47b0b3d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3287, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "doc_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "ref_doc_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "chroma:document": "# SAM\n\n## Overview\n\nSAM (Segment Anything Model) was proposed in [Segment Anything](https://arxiv.org/pdf/2304.02643v1.pdf) by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick.\n\nThe model can be used to predict segmentation masks of any object of interest given an input image. \n\n![example image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/sam-output.png)\n\nThe abstract from the paper is the following:\n\n*We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at [https://segment-anything.com](https://segment-anything.com) to foster research into foundation models for computer vision.*\n\nTips:\n\n- The model predicts binary masks that states the presence or not of the object of interest given an image.\n- The model predicts much better results if input 2D points and/or input bounding boxes are provided\n- You can prompt multiple points for the same image, and predict a single mask. \n- Fine-tuning the model is not supported yet\n- According to the paper, textual input should be also supported. However, at this time of writing this seems to be not supported according to [the official repository](https://github.com/facebookresearch/segment-anything/issues/4#issuecomment-1497626844). \n\n\nThis model was contributed by [ybelkada](https://huggingface.co/ybelkada) and [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/segment-anything).\n\nBelow is an example on how to run mask generation given an image and a 2D point:\n\n```python\nimport torch\nfrom PIL import Image\nimport requests\nfrom transformers import SamModel, SamProcessor\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-huge\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-huge\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\ninput_points = [[[450, 600]]]  # 2D location of a window in the image\n\ninputs = processor(raw_image, input_points=input_points, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmasks = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu()\n)\nscores = outputs.iou_scores\n```\n\nYou can also process your own masks alongside the input images in the processor to be passed to the model."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: bd3f9099-a39e-4eb1-98be-cf48ffaaf83c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"bd3f9099-a39e-4eb1-98be-cf48ffaaf83c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3835a32e-c2af-46e1-ba3c-6380906746cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e7d81cb44b28241fb97cecd8dd189527f3c1d841c6746057893dfc87956a47fe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"def94f09-5597-45a5-85ef-6aaab359301b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"4240c60a84b9cdae77d1526bd7e05c29f24a85b4a06428ceb183d2717392cea1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"51a82e7f-497a-402a-b844-32e634bd52c1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"02f618440a7997e331ddc8f45c1cc1198aafa966f5f0a606ed8e6ebea7021633\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2014, \"end_char_idx\": 5003, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "doc_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "ref_doc_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "chroma:document": "This model was contributed by [ybelkada](https://huggingface.co/ybelkada) and [ArthurZ](https://huggingface.co/ArthurZ).\nThe original code can be found [here](https://github.com/facebookresearch/segment-anything).\n\nBelow is an example on how to run mask generation given an image and a 2D point:\n\n```python\nimport torch\nfrom PIL import Image\nimport requests\nfrom transformers import SamModel, SamProcessor\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-huge\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-huge\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\ninput_points = [[[450, 600]]]  # 2D location of a window in the image\n\ninputs = processor(raw_image, input_points=input_points, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmasks = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu()\n)\nscores = outputs.iou_scores\n```\n\nYou can also process your own masks alongside the input images in the processor to be passed to the model.\n\n```python\nimport torch\nfrom PIL import Image\nimport requests\nfrom transformers import SamModel, SamProcessor\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = SamModel.from_pretrained(\"facebook/sam-vit-huge\").to(device)\nprocessor = SamProcessor.from_pretrained(\"facebook/sam-vit-huge\")\n\nimg_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nraw_image = Image.open(requests.get(img_url, stream=True).raw).convert(\"RGB\")\nmask_url = \"https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png\"\nsegmentation_map = Image.open(requests.get(mask_url, stream=True).raw).convert(\"1\")\ninput_points = [[[450, 600]]]  # 2D location of a window in the image\n\ninputs = processor(raw_image, input_points=input_points, segmentation_maps=segmentation_map, return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nmasks = processor.image_processor.post_process_masks(\n    outputs.pred_masks.cpu(), inputs[\"original_sizes\"].cpu(), inputs[\"reshaped_input_sizes\"].cpu()\n)\nscores = outputs.iou_scores\n```\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SAM.\n\n- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb) for using the model.\n- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) for using the automatic mask generation pipeline.\n- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb) for inference with MedSAM, a fine-tuned version of SAM on the medical domain."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 51a82e7f-497a-402a-b844-32e634bd52c1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/sam/", "title": "SAM", "tokens": 1817, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"51a82e7f-497a-402a-b844-32e634bd52c1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3835a32e-c2af-46e1-ba3c-6380906746cf\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"e7d81cb44b28241fb97cecd8dd189527f3c1d841c6746057893dfc87956a47fe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd3f9099-a39e-4eb1-98be-cf48ffaaf83c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/sam/\", \"title\": \"SAM\", \"tokens\": 1817, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"2e6d0638a44b0713f16ed7ee029283822fa0d928a56d099aacd30dccefbb77e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4491, \"end_char_idx\": 6712, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "doc_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "ref_doc_id": "3835a32e-c2af-46e1-ba3c-6380906746cf", "chroma:document": "- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/segment_anything.ipynb) for using the model.\n- [Demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/automatic_mask_generation.ipynb) for using the automatic mask generation pipeline.\n- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb) for inference with MedSAM, a fine-tuned version of SAM on the medical domain. \ud83c\udf0e\n- [Demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb) for fine-tuning the model on custom data. \ud83c\udf0e\n\n## SlimSAM\n\nSlimSAM, a pruned version of SAM, was proposed in [0.1% Data Makes Segment Anything Slim](https://arxiv.org/abs/2312.05284) by Zigeng Chen et al. SlimSAM reduces the size of the SAM models considerably while maintaining the same performance.\n\nCheckpoints can be found on the [hub](https://huggingface.co/models?other=slimsam), and they can be used as a drop-in replacement of SAM.\n\n## Grounded SAM\n\nOne can combine [Grounding DINO](grounding-dino) with SAM for text-based mask generation as introduced in [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159). You can refer to this [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Grounding%20DINO/GroundingDINO_with_Segment_Anything.ipynb) \ud83c\udf0d for details.\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/grounded_sam.png\"\nalt=\"drawing\" width=\"900\"/>\n\n<small> Grounded SAM overview. Taken from the <a href=\"https://github.com/IDEA-Research/Grounded-Segment-Anything\">original repository</a>. </small>\n\n## SamConfig\n\n[[autodoc]] SamConfig\n\n## SamVisionConfig\n\n[[autodoc]] SamVisionConfig\n\n## SamMaskDecoderConfig\n\n[[autodoc]] SamMaskDecoderConfig\n\n## SamPromptEncoderConfig\n\n[[autodoc]] SamPromptEncoderConfig\n\n\n## SamProcessor\n\n[[autodoc]] SamProcessor\n\n\n## SamImageProcessor\n\n[[autodoc]] SamImageProcessor\n\n\n## SamModel\n\n[[autodoc]] SamModel\n    - forward\n\n\n## TFSamModel\n\n[[autodoc]] TFSamModel\n    - call"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: b7ee07d4-f8d1-488b-a940-9af2bb0ca127
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/falcon/", "title": "Falcon", "tokens": 708, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"b7ee07d4-f8d1-488b-a940-9af2bb0ca127\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon/\", \"title\": \"Falcon\", \"tokens\": 708, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"94b002ce-2c4f-45d6-b699-88faa6d2b303\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/falcon/\", \"title\": \"Falcon\", \"tokens\": 708, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"46b417e980e249b2c735af5b0b39c6ef427a9374b53ec91433f4e8ef45b71464\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "94b002ce-2c4f-45d6-b699-88faa6d2b303", "doc_id": "94b002ce-2c4f-45d6-b699-88faa6d2b303", "ref_doc_id": "94b002ce-2c4f-45d6-b699-88faa6d2b303", "chroma:document": "# Falcon\n\n## Overview\n\nFalcon is a class of causal decoder-only models built by [TII](https://www.tii.ae/). The largest Falcon checkpoints\nhave been trained on >=1T tokens of text, with a particular emphasis on the [RefinedWeb](https://arxiv.org/abs/2306.01116)\ncorpus. They are made available under the Apache 2.0 license.\n\n\nFalcon's architecture is modern and optimized for inference, with multi-query attention and support for efficient\nattention variants like `FlashAttention`. Both 'base' models trained only as causal language models as well as\n'instruct' models that have received further fine-tuning are available.\n\n\nFalcon models are (as of 2023) some of the largest and most powerful open-source language models,\nand consistently rank highly in the [OpenLLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n\n## Converting custom checkpoints \n\n<Tip>\n\nFalcon models were initially added to the Hugging Face Hub as custom code checkpoints. However, Falcon is now fully\nsupported in the Transformers library. If you fine-tuned a model from a custom code checkpoint, we recommend converting\nyour checkpoint to the new in-library format, as this should give significant improvements to stability and\nperformance, especially for generation, as well as removing the need to use `trust_remote_code=True`!\n\n</Tip>\n\nYou can convert custom code checkpoints to full Transformers checkpoints using the `convert_custom_code_checkpoint.py` \nscript located in the\n[Falcon model directory](https://github.com/huggingface/transformers/tree/main/src/transformers/models/falcon)\nof the Transformers library. To use this script, simply call it with \n`python convert_custom_code_checkpoint.py --checkpoint_dir my_model`. This will convert your checkpoint in-place, and\nyou can immediately load it from the directory afterwards with e.g. `from_pretrained()`. If your model hasn't been\nuploaded to the Hub, we recommend making a backup before attempting the conversion, just in case!\n\n\n## FalconConfig\n\n[[autodoc]] FalconConfig\n    - all\n\n## FalconModel\n\n[[autodoc]] FalconModel\n    - forward\n\n## FalconForCausalLM\n\n[[autodoc]] FalconForCausalLM\n    - forward\n\n## FalconForSequenceClassification\n\n[[autodoc]] FalconForSequenceClassification\n    - forward\n\n## FalconForTokenClassification\n\n[[autodoc]] FalconForTokenClassification\n    - forward\n\n## FalconForQuestionAnswering\n\n[[autodoc]] FalconForQuestionAnswering\n    - forward"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 83fa7b51-dda6-4992-be6c-cf7d324bd4cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/barthez/", "title": "BARThez", "tokens": 728, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"83fa7b51-dda6-4992-be6c-cf7d324bd4cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/barthez/\", \"title\": \"BARThez\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"812fe208-0d04-41f1-95dc-4e6c34559978\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/barthez/\", \"title\": \"BARThez\", \"tokens\": 728, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"bca1383888182f628b8ad00e6f0a808ed6b18611ed9ca8461ee8f2bd120cc488\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2286, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "812fe208-0d04-41f1-95dc-4e6c34559978", "doc_id": "812fe208-0d04-41f1-95dc-4e6c34559978", "ref_doc_id": "812fe208-0d04-41f1-95dc-4e6c34559978", "chroma:document": "# BARThez\n\n## Overview\n\nThe BARThez model was proposed in [BARThez: a Skilled Pretrained French Sequence-to-Sequence Model](https://arxiv.org/abs/2010.12321) by Moussa Kamal Eddine, Antoine J.-P. Tixier, Michalis Vazirgiannis on 23 Oct,\n2020.\n\nThe abstract of the paper:\n\n\n*Inductive transfer learning, enabled by self-supervised learning, have taken the entire Natural Language Processing\n(NLP) field by storm, with models such as BERT and BART setting new state of the art on countless natural language\nunderstanding tasks. While there are some notable exceptions, most of the available models and research have been\nconducted for the English language. In this work, we introduce BARThez, the first BART model for the French language\n(to the best of our knowledge). BARThez was pretrained on a very large monolingual French corpus from past research\nthat we adapted to suit BART's perturbation schemes. Unlike already existing BERT-based French language models such as\nCamemBERT and FlauBERT, BARThez is particularly well-suited for generative tasks, since not only its encoder but also\nits decoder is pretrained. In addition to discriminative tasks from the FLUE benchmark, we evaluate BARThez on a novel\nsummarization dataset, OrangeSum, that we release with this paper. We also continue the pretraining of an already\npretrained multilingual BART on BARThez's corpus, and we show that the resulting model, which we call mBARTHez,\nprovides a significant boost over vanilla BARThez, and is on par with or outperforms CamemBERT and FlauBERT.*\n\nThis model was contributed by [moussakam](https://huggingface.co/moussakam). The Authors' code can be found [here](https://github.com/moussaKam/BARThez).\n\n<Tip> \n\nBARThez implementation is the same as BART, except for tokenization. Refer to [BART documentation](bart) for information on \nconfiguration classes and their parameters. BARThez-specific tokenizers are documented below.  \n\n</Tip>\n\n## Resources\n\n- BARThez can be fine-tuned on sequence-to-sequence tasks in a similar way as BART, check:\n  [examples/pytorch/summarization/](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization/README.md).\n\n\n## BarthezTokenizer\n\n[[autodoc]] BarthezTokenizer\n\n## BarthezTokenizerFast\n\n[[autodoc]] BarthezTokenizerFast"}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 1e7068fe-95c6-43b6-9e12-a07250e506a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"1e7068fe-95c6-43b6-9e12-a07250e506a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke/\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ad9af4-fa88-4b27-9d93-13ea18c24a47\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke/\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"12748c3341bcd02d65289074a4d482070f3abc0ed642718db9f9c7aecb7157c6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"395d2acd-0251-4e03-9db7-a4e25b0b1beb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"71230b91dfc3590aa82329aa8db5865bdc3b280da566a97ddf023e647cce60ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3455, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "doc_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "ref_doc_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "chroma:document": "# LUKE\n\n## Overview\n\nThe LUKE model was proposed in [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://arxiv.org/abs/2010.01057) by Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda and Yuji Matsumoto.\nIt is based on RoBERTa and adds entity embeddings as well as an entity-aware self-attention mechanism, which helps\nimprove performance on various downstream tasks involving reasoning about entities such as named entity recognition,\nextractive and cloze-style question answering, entity typing, and relation classification.\n\nThe abstract from the paper is the following:\n\n*Entity representations are useful in natural language tasks involving entities. In this paper, we propose new\npretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed\nmodel treats words and entities in a given text as independent tokens, and outputs contextualized representations of\nthem. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves\npredicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also\npropose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the\ntransformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model\nachieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains\nstate-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification),\nCoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question\nanswering).*\n\nThis model was contributed by [ikuyamada](https://huggingface.co/ikuyamada) and [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/studio-ousia/luke).\n\n## Usage tips\n\n- This implementation is the same as [`RobertaModel`] with the addition of entity embeddings as well\n  as an entity-aware self-attention mechanism, which improves performance on tasks involving reasoning about entities.\n- LUKE treats entities as input tokens; therefore, it takes `entity_ids`, `entity_attention_mask`,\n  `entity_token_type_ids` and `entity_position_ids` as extra input. You can obtain those using\n  [`LukeTokenizer`].\n- [`LukeTokenizer`] takes `entities` and `entity_spans` (character-based start and end\n  positions of the entities in the input text) as extra input. `entities` typically consist of [MASK] entities or\n  Wikipedia entities. The brief description when inputting these entities are as follows:\n\n  - *Inputting [MASK] entities to compute entity representations*: The [MASK] entity is used to mask entities to be\n    predicted during pretraining. When LUKE receives the [MASK] entity, it tries to predict the original entity by\n    gathering the information about the entity from the input text. Therefore, the [MASK] entity can be used to address\n    downstream tasks requiring the information of entities in text such as entity typing, relation classification, and\n    named entity recognition.\n  - *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information\n    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding."}


created_at: 2024-07-27 21:51:38
topic: persistent://default/default/75f2c05a-8be8-4722-bb1a-d317079d4e8a
id: 395d2acd-0251-4e03-9db7-a4e25b0b1beb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/luke/", "title": "LUKE", "tokens": 2104, "retrieve_doc": true, "source": "HF_Transformers", "_node_content": "{\"id_\": \"395d2acd-0251-4e03-9db7-a4e25b0b1beb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke/\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ad9af4-fa88-4b27-9d93-13ea18c24a47\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke/\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"12748c3341bcd02d65289074a4d482070f3abc0ed642718db9f9c7aecb7157c6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1e7068fe-95c6-43b6-9e12-a07250e506a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/luke/\", \"title\": \"LUKE\", \"tokens\": 2104, \"retrieve_doc\": true, \"source\": \"HF_Transformers\"}, \"hash\": \"6601764a885ca8f7100eae75886c48c9d4d81b89bf33a528db84237ee64e9ab9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7d66a52-78c9-4734-ad5a-c43745556fec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5eefc557c728e86ebab5a622ad313d9972541c66d04c8d7d929cac4455b09d29\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1780, \"end_char_idx\": 5005, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "doc_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "ref_doc_id": "d7ad9af4-fa88-4b27-9d93-13ea18c24a47", "chroma:document": "*\n\nThis model was contributed by [ikuyamada](https://huggingface.co/ikuyamada) and [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/studio-ousia/luke).\n\n## Usage tips\n\n- This implementation is the same as [`RobertaModel`] with the addition of entity embeddings as well\n  as an entity-aware self-attention mechanism, which improves performance on tasks involving reasoning about entities.\n- LUKE treats entities as input tokens; therefore, it takes `entity_ids`, `entity_attention_mask`,\n  `entity_token_type_ids` and `entity_position_ids` as extra input. You can obtain those using\n  [`LukeTokenizer`].\n- [`LukeTokenizer`] takes `entities` and `entity_spans` (character-based start and end\n  positions of the entities in the input text) as extra input. `entities` typically consist of [MASK] entities or\n  Wikipedia entities. The brief description when inputting these entities are as follows:\n\n  - *Inputting [MASK] entities to compute entity representations*: The [MASK] entity is used to mask entities to be\n    predicted during pretraining. When LUKE receives the [MASK] entity, it tries to predict the original entity by\n    gathering the information about the entity from the input text. Therefore, the [MASK] entity can be used to address\n    downstream tasks requiring the information of entities in text such as entity typing, relation classification, and\n    named entity recognition.\n  - *Inputting Wikipedia entities to compute knowledge-enhanced token representations*: LUKE learns rich information\n    (or knowledge) about Wikipedia entities during pretraining and stores the information in its entity embedding. By\n    using Wikipedia entities as input tokens, LUKE outputs token representations enriched by the information stored in\n    the embeddings of these entities. This is particularly effective for tasks requiring real-world knowledge, such as\n    question answering.\n\n- There are three head models for the former use case:\n\n  - [`LukeForEntityClassification`], for tasks to classify a single entity in an input text such as\n    entity typing, e.g. the [Open Entity dataset](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html).\n    This model places a linear head on top of the output entity representation.\n  - [`LukeForEntityPairClassification`], for tasks to classify the relationship between two entities\n    such as relation classification, e.g. the [TACRED dataset](https://nlp.stanford.edu/projects/tacred/). This\n    model places a linear head on top of the concatenated output representation of the pair of given entities.\n  - [`LukeForEntitySpanClassification`], for tasks to classify the sequence of entity spans, such as\n    named entity recognition (NER). This model places a linear head on top of the output entity representations. You\n    can address NER using this model by inputting all possible entity spans in the text to the model.\n\n  [`LukeTokenizer`] has a `task` argument, which enables you to easily create an input to these\n  head models by specifying `task=\"entity_classification\"`, `task=\"entity_pair_classification\"`, or\n  `task=\"entity_span_classification\"`. Please refer to the example code of each head models."}


