# embeddings_queue

created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9ad8445e-8663-470e-9dba-faca9c1529a3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9ad8445e-8663-470e-9dba-faca9c1529a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3abe3659-a727-4f58-82b3-d40632ef1606\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ee32d43-3112-4254-9936-06967d947adc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6c7ca7c06a69ee86f4ded253b6157eff87c17477f37162bf326facaad7ff121a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ead0b53-51de-4174-87da-5d8c8e09ad72\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b45673d818dcab6095c8cf5d18d9ee8bf386c86b1fc8c27ec942c6e498d0fc95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5897, \"end_char_idx\": 8335, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "doc_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "ref_doc_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "chroma:document": "\"Banana\"), the tokenizer does not prepend the prefix space to the string.\n\n- When using Flash Attention 2 via `attn_implementation=\"flash_attention_2\"`, don't pass `torch_dtype` to the `from_pretrained` class method and use Automatic Mixed-Precision training. When using `Trainer`, it is simply specifying either `fp16` or `bf16` to `True`. Otherwise, make sure you are using `torch.autocast`. This is required because the Flash Attention only support `fp16` and `bf16` data type.\n\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with LLaMA2. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n- [Llama 2 is here - get it on Hugging Face](https://huggingface.co/blog/llama2), a blog post about Llama 2 and how to use it with \ud83e\udd17 Transformers and \ud83e\udd17 PEFT.\n- [LLaMA 2 - Every Resource you need](https://www.philschmid.de/llama-2), a compilation of relevant resources to learn about LLaMA 2 and how to get started quickly.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1PEQyJO1-f6j0S_XJ8DV50NkpzasXkrzd?usp=sharing) on how to fine-tune Llama 2 in Google Colab using QLoRA and 4-bit precision. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing) on how to fine-tune the \"Llama-v2-7b-guanaco\" model with 4-bit QLoRA and generate Q&A datasets from PDFs. \ud83c\udf0e\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) on how to fine-tune the Llama 2 model with QLoRa, TRL, and Korean text classification dataset. \ud83c\udf0e\ud83c\uddf0\ud83c\uddf7\n\n\u2697\ufe0f Optimization\n- [Fine-tune Llama 2 with DPO](https://huggingface.co/blog/dpo-trl), a guide to using the TRL library's DPO method to fine tune Llama 2 on a specific dataset.\n- [Extended Guide: Instruction-tune Llama 2](https://www.philschmid.de/instruction-tune-llama-2), a guide to training Llama 2 to generate instructions from inputs, transforming the model from instruction-following to instruction-giving.\n- A [notebook](https://colab.research.google.com/drive/1SYpgFpcmtIUzdE7pxqknrM4ArCASfkFQ?usp=sharing) on how to fine-tune the Llama 2 model on a personal computer using QLoRa and TRL."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4ead0b53-51de-4174-87da-5d8c8e09ad72
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4ead0b53-51de-4174-87da-5d8c8e09ad72\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3abe3659-a727-4f58-82b3-d40632ef1606\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ad8445e-8663-470e-9dba-faca9c1529a3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"85fcd5cf2402c08b303ffc3b0e01e8730fb4080b38e2b5634396706f6dcba701\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7c60ede7-cc89-4143-b7fd-2347e718eac7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c18d822fb17a7cd8885b892614de55873e6ce1227443d22b527ff2548f95a187\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7278, \"end_char_idx\": 9606, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "doc_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "ref_doc_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "chroma:document": "\ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing) on how to fine-tune the \"Llama-v2-7b-guanaco\" model with 4-bit QLoRA and generate Q&A datasets from PDFs. \ud83c\udf0e\n\n<PipelineTag pipeline=\"text-classification\"/>\n\n- A [notebook](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) on how to fine-tune the Llama 2 model with QLoRa, TRL, and Korean text classification dataset. \ud83c\udf0e\ud83c\uddf0\ud83c\uddf7\n\n\u2697\ufe0f Optimization\n- [Fine-tune Llama 2 with DPO](https://huggingface.co/blog/dpo-trl), a guide to using the TRL library's DPO method to fine tune Llama 2 on a specific dataset.\n- [Extended Guide: Instruction-tune Llama 2](https://www.philschmid.de/instruction-tune-llama-2), a guide to training Llama 2 to generate instructions from inputs, transforming the model from instruction-following to instruction-giving.\n- A [notebook](https://colab.research.google.com/drive/1SYpgFpcmtIUzdE7pxqknrM4ArCASfkFQ?usp=sharing) on how to fine-tune the Llama 2 model on a personal computer using QLoRa and TRL. \ud83c\udf0e\n\n\u26a1\ufe0f Inference\n- A [notebook](https://colab.research.google.com/drive/1TC56ArKerXUpbgRy5vM3woRsbTEVNq7h?usp=sharing) on how to quantize the Llama 2 model using GPTQ from the AutoGPTQ library. \ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1X1z9Q6domMKl2CnEM0QGHNwidLfR4dW2?usp=sharing) on how to run the Llama 2 Chat Model with 4-bit quantization on a local computer or Google Colab. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- [Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama2-qlora), a complete guide from setup to QLoRA fine-tuning and deployment on Amazon SageMaker.\n- [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm), a guide on using Hugging Face's LLM DLC container for secure and scalable deployment.\n\n\n## LlamaConfig\n\n[[autodoc]] LlamaConfig\n\n\n## LlamaTokenizer\n\n[[autodoc]] LlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LlamaTokenizerFast\n\n[[autodoc]] LlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## LlamaModel\n\n[[autodoc]] LlamaModel\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7c60ede7-cc89-4143-b7fd-2347e718eac7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/llama2", "title": "Llama2", "tokens": 2821, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7c60ede7-cc89-4143-b7fd-2347e718eac7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3abe3659-a727-4f58-82b3-d40632ef1606\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"cecb3790553eb6fa83eea1a4a235226fbb77f97cd3582dd0df28d7b74a7fcfbe\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ead0b53-51de-4174-87da-5d8c8e09ad72\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/llama2\", \"title\": \"Llama2\", \"tokens\": 2821, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a0c5204ad5ba8b1b4a22fbe7cfd354e401328c75a609689400d0fd0e0867a859\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8529, \"end_char_idx\": 9765, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "doc_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "ref_doc_id": "3abe3659-a727-4f58-82b3-d40632ef1606", "chroma:document": "\ud83c\udf0e\n- A [notebook](https://colab.research.google.com/drive/1X1z9Q6domMKl2CnEM0QGHNwidLfR4dW2?usp=sharing) on how to run the Llama 2 Chat Model with 4-bit quantization on a local computer or Google Colab. \ud83c\udf0e\n\n\ud83d\ude80 Deploy\n- [Fine-tune LLaMA 2 (7-70B) on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama2-qlora), a complete guide from setup to QLoRA fine-tuning and deployment on Amazon SageMaker.\n- [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm), a guide on using Hugging Face's LLM DLC container for secure and scalable deployment.\n\n\n## LlamaConfig\n\n[[autodoc]] LlamaConfig\n\n\n## LlamaTokenizer\n\n[[autodoc]] LlamaTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LlamaTokenizerFast\n\n[[autodoc]] LlamaTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - update_post_processor\n    - save_vocabulary\n\n## LlamaModel\n\n[[autodoc]] LlamaModel\n    - forward\n\n\n## LlamaForCausalLM\n\n[[autodoc]] LlamaForCausalLM\n    - forward\n\n## LlamaForSequenceClassification\n\n[[autodoc]] LlamaForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3186853e-cb3a-45ee-aa33-86f0d7ed64c7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mctct", "title": "M-CTC-T", "tokens": 810, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3186853e-cb3a-45ee-aa33-86f0d7ed64c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mctct\", \"title\": \"M-CTC-T\", \"tokens\": 810, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0a25eba-6b2a-4c76-99df-7340cb06f8e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mctct\", \"title\": \"M-CTC-T\", \"tokens\": 810, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d06cfad9abf448714c75a66c975b0c352fb67020b779f3ab6bf196e7d757ef28\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2593, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0a25eba-6b2a-4c76-99df-7340cb06f8e2", "doc_id": "e0a25eba-6b2a-4c76-99df-7340cb06f8e2", "ref_doc_id": "e0a25eba-6b2a-4c76-99df-7340cb06f8e2", "chroma:document": "# M-CTC-T\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, so we won't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.30.0.\nYou can do so by running the following command: `pip install -U transformers==4.30.0`.\n\n</Tip>\n\n## Overview\n\nThe M-CTC-T model was proposed in [Pseudo-Labeling For Massively Multilingual Speech Recognition](https://arxiv.org/abs/2111.00161) by Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. The model is a 1B-param transformer encoder, with a CTC head over 8065 character labels and a language identification head over 60 language ID labels. It is trained on Common Voice (version 6.1, December 2020 release) and VoxPopuli. After training on Common Voice and VoxPopuli, the model is trained on Common Voice only. The labels are unnormalized character-level transcripts (punctuation and capitalization are not removed). The model takes as input Mel filterbank features from a 16Khz audio signal.\n\nThe abstract from the paper is the following:\n\n*Semi-supervised learning through pseudo-labeling has become a staple of state-of-the-art monolingual\nspeech recognition systems. In this work, we extend pseudo-labeling to massively multilingual speech\nrecognition with 60 languages. We propose a simple pseudo-labeling recipe that works well even\nwith low-resource languages: train a supervised multilingual model, fine-tune it with semi-supervised\nlearning on a target language, generate pseudo-labels for that language, and train a final model using\npseudo-labels for all languages, either from scratch or by fine-tuning. Experiments on the labeled\nCommon Voice and unlabeled VoxPopuli datasets show that our recipe can yield a model with better\nperformance for many languages that also transfers well to LibriSpeech.*\n\nThis model was contributed by [cwkeam](https://huggingface.co/cwkeam). The original code can be found [here](https://github.com/flashlight/wav2letter/tree/main/recipes/mling_pl).\n\n## Usage tips\n\nThe PyTorch version of this model is only available in torch 1.9 and higher.\n\n## Resources\n\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## MCTCTConfig\n\n[[autodoc]] MCTCTConfig\n\n## MCTCTFeatureExtractor\n\n[[autodoc]] MCTCTFeatureExtractor\n    - __call__\n\n## MCTCTProcessor\n\n[[autodoc]] MCTCTProcessor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## MCTCTModel\n\n[[autodoc]] MCTCTModel\n    - forward\n\n## MCTCTForCTC\n\n[[autodoc]] MCTCTForCTC\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e9d75eee-7bf8-4ad1-b551-c1b2937f2dc6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e9d75eee-7bf8-4ad1-b551-c1b2937f2dc6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4d175ef3-5330-4eee-951c-0f81e34b92c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3aeb17ea8f7970d90ef023bdec387aeac404219c8eedc74ac9872caa704178\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7f96dca7-ab3f-47c7-ae7c-d41fd3b4fc01\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"def884d2e42ca979f1be2a5d3a151172a9cf63fea68255e68ee098c15f7b3528\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2114, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4d175ef3-5330-4eee-951c-0f81e34b92c6", "doc_id": "4d175ef3-5330-4eee-951c-0f81e34b92c6", "ref_doc_id": "4d175ef3-5330-4eee-951c-0f81e34b92c6", "chroma:document": "# Blenderbot Small\n\nNote that [`BlenderbotSmallModel`] and\n[`BlenderbotSmallForConditionalGeneration`] are only used in combination with the checkpoint\n[facebook/blenderbot-90M](https://huggingface.co/facebook/blenderbot-90M). Larger Blenderbot checkpoints should\ninstead be used with [`BlenderbotModel`] and\n[`BlenderbotForConditionalGeneration`]\n\n## Overview\n\nThe Blender chatbot model was proposed in [Recipes for building an open-domain chatbot](https://arxiv.org/pdf/2004.13637.pdf) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,\nJing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.\n\nThe abstract of the paper is the following:\n\n*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that\nscaling neural models in the number of parameters and the size of the data they are trained on gives improved results,\nwe show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of\nskills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to\ntheir partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent\npersona. We show that large scale models can learn these skills when given appropriate training data and choice of\ngeneration strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models\nand code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn\ndialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing\nfailure cases of our models.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The authors' code can be\nfound [here](https://github.com/facebookresearch/ParlAI).\n\n## Usage tips\n\nBlenderbot Small is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than \nthe left."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7f96dca7-ab3f-47c7-ae7c-d41fd3b4fc01
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/blenderbot-small", "title": "Blenderbot Small", "tokens": 976, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7f96dca7-ab3f-47c7-ae7c-d41fd3b4fc01\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4d175ef3-5330-4eee-951c-0f81e34b92c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd3aeb17ea8f7970d90ef023bdec387aeac404219c8eedc74ac9872caa704178\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9d75eee-7bf8-4ad1-b551-c1b2937f2dc6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/blenderbot-small\", \"title\": \"Blenderbot Small\", \"tokens\": 976, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"10ead889d19a2d2e49f4170d6262633564bbf0be59b9eb0ea1ff4a8bdea56c0d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2117, \"end_char_idx\": 3419, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4d175ef3-5330-4eee-951c-0f81e34b92c6", "doc_id": "4d175ef3-5330-4eee-951c-0f81e34b92c6", "ref_doc_id": "4d175ef3-5330-4eee-951c-0f81e34b92c6", "chroma:document": "## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## BlenderbotSmallConfig\n\n[[autodoc]] BlenderbotSmallConfig\n\n## BlenderbotSmallTokenizer\n\n[[autodoc]] BlenderbotSmallTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## BlenderbotSmallTokenizerFast\n\n[[autodoc]] BlenderbotSmallTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## BlenderbotSmallModel\n\n[[autodoc]] BlenderbotSmallModel\n    - forward\n\n## BlenderbotSmallForConditionalGeneration\n\n[[autodoc]] BlenderbotSmallForConditionalGeneration\n    - forward\n\n## BlenderbotSmallForCausalLM\n\n[[autodoc]] BlenderbotSmallForCausalLM\n    - forward\n\n</pt>\n<tf>\n\n## TFBlenderbotSmallModel\n\n[[autodoc]] TFBlenderbotSmallModel\n    - call\n\n## TFBlenderbotSmallForConditionalGeneration\n\n[[autodoc]] TFBlenderbotSmallForConditionalGeneration\n    - call\n\n</tf>\n<jax>\n\n## FlaxBlenderbotSmallModel\n\n[[autodoc]] FlaxBlenderbotSmallModel\n    - __call__\n    - encode\n    - decode\n\n## FlaxBlenderbotForConditionalGeneration\n\n[[autodoc]] FlaxBlenderbotSmallForConditionalGeneration\n    - __call__\n    - encode\n    - decode\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e1282ccd-bb68-43e3-8d7b-ffcb1994ce9c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/vitmatte", "title": "ViTMatte", "tokens": 784, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e1282ccd-bb68-43e3-8d7b-ffcb1994ce9c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitmatte\", \"title\": \"ViTMatte\", \"tokens\": 784, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3d5b247d-2648-4707-8970-21b91bdcf048\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/vitmatte\", \"title\": \"ViTMatte\", \"tokens\": 784, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"184feb2a440ce292f25cd540ed7243d714a25687e1072b5a03de061eed60f10e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2804, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3d5b247d-2648-4707-8970-21b91bdcf048", "doc_id": "3d5b247d-2648-4707-8970-21b91bdcf048", "ref_doc_id": "3d5b247d-2648-4707-8970-21b91bdcf048", "chroma:document": "# ViTMatte\n\n## Overview\n\nThe ViTMatte model was proposed in [Boosting Image Matting with Pretrained Plain Vision Transformers](https://arxiv.org/abs/2305.15272) by Jingfeng Yao, Xinggang Wang, Shusheng Yang, Baoyuan Wang.\nViTMatte leverages plain [Vision Transformers](vit) for the task of image matting, which is the process of accurately estimating the foreground object in images and videos.\n\nThe abstract from the paper is the following:\n\n*Recently, plain vision Transformers (ViTs) have shown impressive performance on various computer vision tasks, thanks to their strong modeling capacity and large-scale pretraining. However, they have not yet conquered the problem of image matting. We hypothesize that image matting could also be boosted by ViTs and present a new efficient and robust ViT-based matting system, named ViTMatte. Our method utilizes (i) a hybrid attention mechanism combined with a convolution neck to help ViTs achieve an excellent performance-computation trade-off in matting tasks. (ii) Additionally, we introduce the detail capture module, which just consists of simple lightweight convolutions to complement the detailed information required by matting. To the best of our knowledge, ViTMatte is the first work to unleash the potential of ViT on image matting with concise adaptation. It inherits many superior properties from ViT to matting, including various pretraining strategies, concise architecture design, and flexible inference strategies. We evaluate ViTMatte on Composition-1k and Distinctions-646, the most commonly used benchmark for image matting, our method achieves state-of-the-art performance and outperforms prior matting works by a large margin.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/hustvl/ViTMatte).\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vitmatte_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ViTMatte high-level overview. Taken from the <a href=\"https://arxiv.org/abs/2305.15272\">original paper.</a> </small>\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ViTMatte.\n\n- A demo notebook regarding inference with [`VitMatteForImageMatting`], including background replacement, can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViTMatte).\n\n<Tip>\n\nThe model expects both the image and trimap (concatenated) as input. Use [`ViTMatteImageProcessor`] for this purpose.\n</Tip>\n\n## VitMatteConfig\n\n[[autodoc]] VitMatteConfig\n\n## VitMatteImageProcessor\n\n[[autodoc]] VitMatteImageProcessor\n    - preprocess\n\n## VitMatteForImageMatting\n\n[[autodoc]] VitMatteForImageMatting\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 34843729-1817-4eaa-b045-0b57e3c8b839
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"34843729-1817-4eaa-b045-0b57e3c8b839\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"40f1a199-bb43-4013-92e3-3138c60948a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"67053795ea1275807081df9509ab2419d31b1e98b4ee64704806c04ebacf4cac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b88ab02-5889-45b5-bf9c-26a7d3b1b701\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d54478e8ce727dbf775c2dc7b72a0970d59cd293bb6020770cc330ad26a7074e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2486, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "40f1a199-bb43-4013-92e3-3138c60948a3", "doc_id": "40f1a199-bb43-4013-92e3-3138c60948a3", "ref_doc_id": "40f1a199-bb43-4013-92e3-3138c60948a3", "chroma:document": "# BLOOM\n\n## Overview\n\nThe BLOOM model has been proposed with its various versions through the [BigScience Workshop](https://bigscience.huggingface.co/). BigScience is inspired by other open science initiatives where researchers have pooled their time and resources to collectively achieve a higher impact.\nThe architecture of BLOOM is essentially similar to GPT3 (auto-regressive model for next token prediction), but has been trained on 46 different languages and 13 programming languages.\nSeveral smaller versions of the models have been trained on the same dataset. BLOOM is available in the following versions:\n\n- [bloom-560m](https://huggingface.co/bigscience/bloom-560m)\n- [bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)\n- [bloom-1b7](https://huggingface.co/bigscience/bloom-1b7)\n- [bloom-3b](https://huggingface.co/bigscience/bloom-3b)\n- [bloom-7b1](https://huggingface.co/bigscience/bloom-7b1)\n- [bloom](https://huggingface.co/bigscience/bloom) (176B parameters)\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with BLOOM. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n<PipelineTag pipeline=\"text-generation\"/>\n\n- [`BloomForCausalLM`] is supported by this [causal language modeling example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling#gpt-2gpt-and-causal-language-modeling) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb).\n\nSee also:\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n\n\n\u26a1\ufe0f Inference\n- A blog on [Optimization story: Bloom inference](https://huggingface.co/blog/bloom-inference-optimization).\n- A blog on [Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate](https://huggingface.co/blog/bloom-inference-pytorch-scripts).\n\n\u2699\ufe0f Training\n- A blog on [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed).\n\n## BloomConfig\n\n[[autodoc]] BloomConfig\n    - all\n\n## BloomTokenizerFast\n\n[[autodoc]] BloomTokenizerFast\n    - all"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6b88ab02-5889-45b5-bf9c-26a7d3b1b701
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/bloom", "title": "BLOOM", "tokens": 954, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6b88ab02-5889-45b5-bf9c-26a7d3b1b701\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"40f1a199-bb43-4013-92e3-3138c60948a3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"67053795ea1275807081df9509ab2419d31b1e98b4ee64704806c04ebacf4cac\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"34843729-1817-4eaa-b045-0b57e3c8b839\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/bloom\", \"title\": \"BLOOM\", \"tokens\": 954, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61e91917fe5664c93e7b4479bc0df5e97a850539af73712d0dc64e0ffc61affb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1992, \"end_char_idx\": 3070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "40f1a199-bb43-4013-92e3-3138c60948a3", "doc_id": "40f1a199-bb43-4013-92e3-3138c60948a3", "ref_doc_id": "40f1a199-bb43-4013-92e3-3138c60948a3", "chroma:document": "\u26a1\ufe0f Inference\n- A blog on [Optimization story: Bloom inference](https://huggingface.co/blog/bloom-inference-optimization).\n- A blog on [Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate](https://huggingface.co/blog/bloom-inference-pytorch-scripts).\n\n\u2699\ufe0f Training\n- A blog on [The Technology Behind BLOOM Training](https://huggingface.co/blog/bloom-megatron-deepspeed).\n\n## BloomConfig\n\n[[autodoc]] BloomConfig\n    - all\n\n## BloomTokenizerFast\n\n[[autodoc]] BloomTokenizerFast\n    - all\n\n\n<frameworkcontent>\n<pt>\n\n## BloomModel\n\n[[autodoc]] BloomModel\n    - forward\n\n## BloomForCausalLM\n\n[[autodoc]] BloomForCausalLM\n    - forward\n\n## BloomForSequenceClassification\n\n[[autodoc]] BloomForSequenceClassification\n    - forward\n\n## BloomForTokenClassification\n\n[[autodoc]] BloomForTokenClassification\n    - forward\n\n## BloomForQuestionAnswering\n\n[[autodoc]] BloomForQuestionAnswering\n    - forward\n\n</pt>\n<jax>\n\n## FlaxBloomModel\n\n[[autodoc]] FlaxBloomModel\n    - __call__\n\n## FlaxBloomForCausalLM\n\n[[autodoc]] FlaxBloomForCausalLM\n    - __call__\n\n</jax>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1f03b055-5d3d-4d77-ab90-db22988625fb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1f03b055-5d3d-4d77-ab90-db22988625fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c243227e-f8ed-4382-a40c-062fe05fcc2d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6632ae6f721f2c6cc01e0a517a1126f5aee04fb83808055f9425a7c4d6bf4bf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c9d3f43b-1c8a-4785-8ea0-21355946eb26\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"baa69438b0f3770996ef35a1512dfbe14e961adc2c6696d605b33ddd36bad7d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2755, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c243227e-f8ed-4382-a40c-062fe05fcc2d", "doc_id": "c243227e-f8ed-4382-a40c-062fe05fcc2d", "ref_doc_id": "c243227e-f8ed-4382-a40c-062fe05fcc2d", "chroma:document": "# Speech2Text2\n\n  <Tip warning={true}>\n\n  This model is in maintenance mode only, we don't accept any new PRs changing its code.\n  If you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\n  You can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n  </Tip>\n\n## Overview\n\nThe Speech2Text2 model is used together with [Wav2Vec2](wav2vec2) for Speech Translation models proposed in\n[Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://arxiv.org/abs/2104.06678) by\nChanghan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, Alexis Conneau.\n\nSpeech2Text2 is a *decoder-only* transformer model that can be used with any speech *encoder-only*, such as\n[Wav2Vec2](wav2vec2) or [HuBERT](hubert) for Speech-to-Text tasks. Please refer to the\n[SpeechEncoderDecoder](speech-encoder-decoder) class on how to combine Speech2Text2 with any speech *encoder-only*\nmodel.\n\nThis model was contributed by [Patrick von Platen](https://huggingface.co/patrickvonplaten).\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/blob/1f7ef9ed1e1061f8c7f88f8b94c7186834398690/fairseq/models/wav2vec/wav2vec2_asr.py#L266).\n\n## Usage tips\n\n- Speech2Text2 achieves state-of-the-art results on the CoVoST Speech Translation dataset. For more information, see\n  the [official models](https://huggingface.co/models?other=speech2text2) .\n- Speech2Text2 is always used within the [SpeechEncoderDecoder](speech-encoder-decoder) framework.\n- Speech2Text2's tokenizer is based on [fastBPE](https://github.com/glample/fastBPE).\n\n## Inference\n\nSpeech2Text2's [`SpeechEncoderDecoderModel`] model accepts raw waveform input values from speech and\nmakes use of [`~generation.GenerationMixin.generate`] to translate the input speech\nautoregressively to the target language.\n\nThe [`Wav2Vec2FeatureExtractor`] class is responsible for preprocessing the input speech and\n[`Speech2Text2Tokenizer`] decodes the generated target tokens to the target string. The\n[`Speech2Text2Processor`] wraps [`Wav2Vec2FeatureExtractor`] and\n[`Speech2Text2Tokenizer`] into a single instance to both extract the input features and decode the\npredicted token ids.\n\n- Step-by-step Speech Translation\n\n```python\n>>> import torch\n>>> from transformers import Speech2Text2Processor, SpeechEncoderDecoderModel\n>>> from datasets import load_dataset\n>>> import soundfile as sf\n\n>>> model = SpeechEncoderDecoderModel.from_pretrained(\"facebook/s2t-wav2vec2-large-en-de\")\n>>> processor = Speech2Text2Processor.from_pretrained(\"facebook/s2t-wav2vec2-large-en-de\")\n\n\n>>> def map_to_array(batch):\n...     speech, _ = sf.read(batch[\"file\"])\n...     batch[\"speech\"] = speech\n...     return batch"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c9d3f43b-1c8a-4785-8ea0-21355946eb26
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/speech_to_text_2", "title": "Speech2Text2", "tokens": 1275, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c9d3f43b-1c8a-4785-8ea0-21355946eb26\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c243227e-f8ed-4382-a40c-062fe05fcc2d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6632ae6f721f2c6cc01e0a517a1126f5aee04fb83808055f9425a7c4d6bf4bf\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f03b055-5d3d-4d77-ab90-db22988625fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/speech_to_text_2\", \"title\": \"Speech2Text2\", \"tokens\": 1275, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"172f06374d203cd12fa52539533315d88a0058e164e49fe90134fcc26172b736\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2630, \"end_char_idx\": 4308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c243227e-f8ed-4382-a40c-062fe05fcc2d", "doc_id": "c243227e-f8ed-4382-a40c-062fe05fcc2d", "ref_doc_id": "c243227e-f8ed-4382-a40c-062fe05fcc2d", "chroma:document": ">>> def map_to_array(batch):\n...     speech, _ = sf.read(batch[\"file\"])\n...     batch[\"speech\"] = speech\n...     return batch\n\n\n>>> ds = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> ds = ds.map(map_to_array)\n\n>>> inputs = processor(ds[\"speech\"][0], sampling_rate=16_000, return_tensors=\"pt\")\n>>> generated_ids = model.generate(inputs=inputs[\"input_values\"], attention_mask=inputs[\"attention_mask\"])\n\n>>> transcription = processor.batch_decode(generated_ids)\n```\n\n- Speech Translation via Pipelines\n\n  The automatic speech recognition pipeline can also be used to translate speech in just a couple lines of code\n\n```python\n>>> from datasets import load_dataset\n>>> from transformers import pipeline\n\n>>> librispeech_en = load_dataset(\"hf-internal-testing/librispeech_asr_dummy\", \"clean\", split=\"validation\")\n>>> asr = pipeline(\n...     \"automatic-speech-recognition\",\n...     model=\"facebook/s2t-wav2vec2-large-en-de\",\n...     feature_extractor=\"facebook/s2t-wav2vec2-large-en-de\",\n... )\n\n>>> translation_de = asr(librispeech_en[0][\"file\"])\n```\n\nSee [model hub](https://huggingface.co/models?filter=speech2text2) to look for Speech2Text2 checkpoints.\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## Speech2Text2Config\n\n[[autodoc]] Speech2Text2Config\n\n## Speech2TextTokenizer\n\n[[autodoc]] Speech2Text2Tokenizer\n    - batch_decode\n    - decode\n    - save_vocabulary\n\n## Speech2Text2Processor\n\n[[autodoc]] Speech2Text2Processor\n    - __call__\n    - from_pretrained\n    - save_pretrained\n    - batch_decode\n    - decode\n\n## Speech2Text2ForCausalLM\n\n[[autodoc]] Speech2Text2ForCausalLM\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 793209c9-ce9f-49a2-a0cd-83324438303a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/layoutxlm", "title": "LayoutXLM", "tokens": 832, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"793209c9-ce9f-49a2-a0cd-83324438303a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutxlm\", \"title\": \"LayoutXLM\", \"tokens\": 832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ea381379-d33e-4042-a017-2cb403234502\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/layoutxlm\", \"title\": \"LayoutXLM\", \"tokens\": 832, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a326e71685d22f9abcd2a2790fae4219c29a260e61080542d5a4791e8503639b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2724, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ea381379-d33e-4042-a017-2cb403234502", "doc_id": "ea381379-d33e-4042-a017-2cb403234502", "ref_doc_id": "ea381379-d33e-4042-a017-2cb403234502", "chroma:document": "# LayoutXLM\n\n## Overview\n\nLayoutXLM was proposed in [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836) by Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha\nZhang, Furu Wei. It's a multilingual extension of the [LayoutLMv2 model](https://arxiv.org/abs/2012.14740) trained\non 53 languages.\n\nThe abstract from the paper is the following:\n\n*Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document\nunderstanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In\nthis paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to\nbridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also\nintroduce a multilingual form understanding benchmark dataset named XFUN, which includes form understanding samples in\n7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled\nfor each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA\ncross-lingual pre-trained models on the XFUN dataset.*\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm).\n\n## Usage tips and examples\n\nOne can directly plug in the weights of LayoutXLM into a LayoutLMv2 model, like so:\n\n```python\nfrom transformers import LayoutLMv2Model\n\nmodel = LayoutLMv2Model.from_pretrained(\"microsoft/layoutxlm-base\")\n```\n\nNote that LayoutXLM has its own tokenizer, based on\n[`LayoutXLMTokenizer`]/[`LayoutXLMTokenizerFast`]. You can initialize it as\nfollows:\n\n```python\nfrom transformers import LayoutXLMTokenizer\n\ntokenizer = LayoutXLMTokenizer.from_pretrained(\"microsoft/layoutxlm-base\")\n```\n\nSimilar to LayoutLMv2, you can use [`LayoutXLMProcessor`] (which internally applies\n[`LayoutLMv2ImageProcessor`] and\n[`LayoutXLMTokenizer`]/[`LayoutXLMTokenizerFast`] in sequence) to prepare all\ndata for the model.\n\n<Tip>\n\nAs LayoutXLM's architecture is equivalent to that of LayoutLMv2, one can refer to [LayoutLMv2's documentation page](layoutlmv2) for all tips, code examples and notebooks.\n</Tip>\n\n## LayoutXLMTokenizer\n\n[[autodoc]] LayoutXLMTokenizer\n    - __call__\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## LayoutXLMTokenizerFast\n\n[[autodoc]] LayoutXLMTokenizerFast\n    - __call__\n\n## LayoutXLMProcessor\n\n[[autodoc]] LayoutXLMProcessor\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 953e3230-c655-49eb-91a0-70dd63a86fba
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"953e3230-c655-49eb-91a0-70dd63a86fba\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5359fe09-bc3c-4aea-8291-1dee1df1237d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"70e2ff3d7a29b90e95bb444bfb9561e5670d993bd13785a43893145ab2cf7663\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"75d24270-cb7a-43ed-b6c9-cbbfa536c34d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb6d4636a7e2f8a902f0ba299c3550383d1b19657c4f17f6438af1f26a29487c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2853, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "doc_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "ref_doc_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "chroma:document": "# MegatronBERT\n\n## Overview\n\nThe MegatronBERT model was proposed in [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model\nParallelism](https://arxiv.org/abs/1909.08053) by Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley,\nJared Casper and Bryan Catanzaro.\n\nThe abstract from the paper is the following:\n\n*Recent work in language modeling demonstrates that training large transformer models advances the state of the art in\nNatural Language Processing applications. However, very large models can be quite difficult to train due to memory\nconstraints. In this work, we present our techniques for training very large transformer models and implement a simple,\nefficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our\napproach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model\nparallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We\nillustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain\n15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline\nthat sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, \nit contains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 75d24270-cb7a-43ed-b6c9-cbbfa536c34d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"75d24270-cb7a-43ed-b6c9-cbbfa536c34d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5359fe09-bc3c-4aea-8291-1dee1df1237d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"70e2ff3d7a29b90e95bb444bfb9561e5670d993bd13785a43893145ab2cf7663\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"953e3230-c655-49eb-91a0-70dd63a86fba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d146e6d934b8bbb96048de9cdbc97c858a525477112aed031680e2c7ae88c2e3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"89a5842c-0114-42d2-bbf1-1daac2a27f86\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3cdd11a072a95f91e7e95f426ae5b33469743a147584db1713803c99bf23491c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1337, \"end_char_idx\": 3537, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "doc_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "ref_doc_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "chroma:document": "To demonstrate that large language models can further advance\nthe state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9\nbillion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in\nBERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we\nachieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA\naccuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy\nof 89.4%).*\n\nThis model was contributed by [jdemouth](https://huggingface.co/jdemouth). The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, \nit contains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).\n\nAlternatively, you can directly download the checkpoints using:\n\nBERT-345M-uncased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip\n-O megatron_bert_345m_v0_1_uncased.zip\n```\n\nBERT-345M-cased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O\nmegatron_bert_345m_v0_1_cased.zip\n```\n\nOnce you have obtained the checkpoints from NVIDIA GPU Cloud (NGC), you have to convert them to a format that will\neasily be loaded by Hugging Face Transformers and our port of the BERT code.\n\nThe following commands allow you to do the conversion."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 89a5842c-0114-42d2-bbf1-1daac2a27f86
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/megatron-bert", "title": "MegatronBERT", "tokens": 1498, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"89a5842c-0114-42d2-bbf1-1daac2a27f86\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5359fe09-bc3c-4aea-8291-1dee1df1237d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"70e2ff3d7a29b90e95bb444bfb9561e5670d993bd13785a43893145ab2cf7663\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"75d24270-cb7a-43ed-b6c9-cbbfa536c34d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/megatron-bert\", \"title\": \"MegatronBERT\", \"tokens\": 1498, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fce25c1807d0b9d3d27fc9228a68ccd4ea1a43cf8c7417be381a939d04cd7a71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2071, \"end_char_idx\": 5249, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "doc_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "ref_doc_id": "5359fe09-bc3c-4aea-8291-1dee1df1237d", "chroma:document": "The original code can be found [here](https://github.com/NVIDIA/Megatron-LM). \nThat repository contains a multi-GPU and multi-node implementation of the Megatron Language models. In particular, \nit contains a hybrid model parallel approach using \"tensor parallel\" and \"pipeline parallel\" techniques.\n\n## Usage tips\n\nWe have provided pretrained [BERT-345M](https://ngc.nvidia.com/catalog/models/nvidia:megatron_bert_345m) checkpoints\nfor use to evaluate or finetuning downstream tasks.\n\nTo access these checkpoints, first [sign up](https://ngc.nvidia.com/signup) for and setup the NVIDIA GPU Cloud (NGC)\nRegistry CLI. Further documentation for downloading models can be found in the [NGC documentation](https://docs.nvidia.com/dgx/ngc-registry-cli-user-guide/index.html#topic_6_4_1).\n\nAlternatively, you can directly download the checkpoints using:\n\nBERT-345M-uncased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_uncased/zip\n-O megatron_bert_345m_v0_1_uncased.zip\n```\n\nBERT-345M-cased:\n\n```bash\nwget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/megatron_bert_345m/versions/v0.1_cased/zip -O\nmegatron_bert_345m_v0_1_cased.zip\n```\n\nOnce you have obtained the checkpoints from NVIDIA GPU Cloud (NGC), you have to convert them to a format that will\neasily be loaded by Hugging Face Transformers and our port of the BERT code.\n\nThe following commands allow you to do the conversion. We assume that the folder `models/megatron_bert` contains\n`megatron_bert_345m_v0_1_{cased, uncased}.zip` and that the commands are run from inside that folder:\n\n```bash\npython3 $PATH_TO_TRANSFORMERS/models/megatron_bert/convert_megatron_bert_checkpoint.py megatron_bert_345m_v0_1_uncased.zip\n```\n\n```bash\npython3 $PATH_TO_TRANSFORMERS/models/megatron_bert/convert_megatron_bert_checkpoint.py megatron_bert_345m_v0_1_cased.zip\n```\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## MegatronBertConfig\n\n[[autodoc]] MegatronBertConfig\n\n## MegatronBertModel\n\n[[autodoc]] MegatronBertModel\n    - forward\n\n## MegatronBertForMaskedLM\n\n[[autodoc]] MegatronBertForMaskedLM\n    - forward\n\n## MegatronBertForCausalLM\n\n[[autodoc]] MegatronBertForCausalLM\n    - forward\n\n## MegatronBertForNextSentencePrediction\n\n[[autodoc]] MegatronBertForNextSentencePrediction\n    - forward\n\n## MegatronBertForPreTraining\n\n[[autodoc]] MegatronBertForPreTraining\n    - forward\n\n## MegatronBertForSequenceClassification\n\n[[autodoc]] MegatronBertForSequenceClassification\n    - forward\n\n## MegatronBertForMultipleChoice\n\n[[autodoc]] MegatronBertForMultipleChoice\n    - forward\n\n## MegatronBertForTokenClassification\n\n[[autodoc]] MegatronBertForTokenClassification\n    - forward\n\n## MegatronBertForQuestionAnswering\n\n[[autodoc]] MegatronBertForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6bb44598-4a72-46aa-a79e-6947cdb34a8e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6bb44598-4a72-46aa-a79e-6947cdb34a8e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61bc83ab28435751bdfd4263c41458251e18c4d000493fe39a77d44cf36f56c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"116aa560-0d38-451f-b85d-59f318f022a3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d20385311eda57a0037cf62219d3e6514f0aafad268755f4b951cdd63ddfa2c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 940, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3", "doc_id": "a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3", "ref_doc_id": "a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3", "chroma:document": "# XLM-ProphetNet\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xprophetnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xprophetnet-large-wiki100-cased-xglue-ntg\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign\n@patrickvonplaten"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 116aa560-0d38-451f-b85d-59f318f022a3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet", "title": "XLM-ProphetNet", "tokens": 1060, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"116aa560-0d38-451f-b85d-59f318f022a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61bc83ab28435751bdfd4263c41458251e18c4d000493fe39a77d44cf36f56c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6bb44598-4a72-46aa-a79e-6947cdb34a8e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-prophetnet\", \"title\": \"XLM-ProphetNet\", \"tokens\": 1060, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"aaf2fc33b13f2830513c5d739c2015f8e2897f5a79266f6a8445d9c820803959\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3442, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3", "doc_id": "a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3", "ref_doc_id": "a01ea479-b7e8-4fdc-b651-4ec4c2b84fc3", "chroma:document": "# XLM-ProphetNet\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=xprophetnet\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-xprophetnet-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/xprophetnet-large-wiki100-cased-xglue-ntg\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n**DISCLAIMER:** If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title) and assign\n@patrickvonplaten\n\n\n## Overview\n\nThe XLM-ProphetNet model was proposed in [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei\nZhang, Ming Zhou on 13 Jan, 2020.\n\nXLM-ProphetNet is an encoder-decoder model and can predict n-future tokens for \"ngram\" language modeling instead of\njust the next token. Its architecture is identical to ProhpetNet, but the model was trained on the multi-lingual\n\"wiki100\" Wikipedia dump. XLM-ProphetNet's model architecture and pretraining objective is same as ProphetNet, but XLM-ProphetNet was pre-trained on the cross-lingual dataset XGLUE.\n\nThe abstract from the paper is the following:\n\n*In this paper, we present a new sequence-to-sequence pretraining model called ProphetNet, which introduces a novel\nself-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of\nthe optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by\nn-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time\nstep. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent\noverfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale\ndataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for\nabstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new\nstate-of-the-art results on all these datasets compared to the models using the same scale pretraining corpus.*\n\nThe Authors' code can be found [here](https://github.com/microsoft/ProphetNet).\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Translation task guide](../tasks/translation)\n- [Summarization task guide](../tasks/summarization)\n\n## XLMProphetNetConfig\n\n[[autodoc]] XLMProphetNetConfig\n\n## XLMProphetNetTokenizer\n\n[[autodoc]] XLMProphetNetTokenizer\n\n## XLMProphetNetModel\n\n[[autodoc]] XLMProphetNetModel\n\n## XLMProphetNetEncoder\n\n[[autodoc]] XLMProphetNetEncoder\n\n## XLMProphetNetDecoder\n\n[[autodoc]] XLMProphetNetDecoder\n\n## XLMProphetNetForConditionalGeneration\n\n[[autodoc]] XLMProphetNetForConditionalGeneration\n\n## XLMProphetNetForCausalLM\n\n[[autodoc]] XLMProphetNetForCausalLM"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 88d29fba-41ab-4623-bb90-dcc23f1e1301
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/open-llama", "title": "Open-Llama", "tokens": 523, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"88d29fba-41ab-4623-bb90-dcc23f1e1301\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/open-llama\", \"title\": \"Open-Llama\", \"tokens\": 523, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"410baf30-0493-4653-8f58-d4e07ba52f73\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/open-llama\", \"title\": \"Open-Llama\", \"tokens\": 523, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5942ea5a19e1f1c5703b3a327baecda326badebb76f53fc3451b5b56b7e3d84a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1418, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "410baf30-0493-4653-8f58-d4e07ba52f73", "doc_id": "410baf30-0493-4653-8f58-d4e07ba52f73", "ref_doc_id": "410baf30-0493-4653-8f58-d4e07ba52f73", "chroma:document": "# Open-Llama\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\n\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.31.0.\nYou can do so by running the following command: `pip install -U transformers==4.31.0`.\n\n</Tip>\n\n<Tip warning={true}>\n\nThis model differs from the [OpenLLaMA models](https://huggingface.co/models?search=openllama) on the Hugging Face Hub, which primarily use the [LLaMA](llama) architecture.\n\n</Tip>\n\n## Overview\n\nThe Open-Llama model was proposed in the open source Open-Llama project by community developer s-JoL.\n\nThe model is mainly based on LLaMA with some modifications, incorporating memory-efficient attention from Xformers, stable embedding from Bloom, and shared input-output embedding from PaLM.\nAnd the model is pre-trained on both Chinese and English, which gives it better performance on Chinese language tasks.\n\nThis model was contributed by [s-JoL](https://huggingface.co/s-JoL).\nThe original code was released on GitHub by [s-JoL](https://github.com/s-JoL), but is now removed.\n\n## OpenLlamaConfig\n\n[[autodoc]] OpenLlamaConfig\n\n## OpenLlamaModel\n\n[[autodoc]] OpenLlamaModel\n    - forward\n\n## OpenLlamaForCausalLM\n\n[[autodoc]] OpenLlamaForCausalLM\n    - forward\n\n## OpenLlamaForSequenceClassification\n\n[[autodoc]] OpenLlamaForSequenceClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7085f72a-ee7b-43e9-8331-4facb0ebd895
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7085f72a-ee7b-43e9-8331-4facb0ebd895\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f9ff5f3-b9c7-48e0-a4a0-046ca8500944\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e9cf2ca3-5513-459d-a455-1bf9c10be388\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"74284ebc51ce0dc77778d818d996671bd4a235357803d538f2f8e275cfd84d97\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2863, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "ref_doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "chroma:document": "# Phi\n\n## Overview\n\nThe Phi-1 model was proposed in [Textbooks Are All You Need](https://arxiv.org/abs/2306.11644) by Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C\u00e9sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S\u00e9bastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee and Yuanzhi Li.\n\nThe Phi-1.5 model was proposed in [Textbooks Are All You Need II: phi-1.5 technical report](https://arxiv.org/abs/2309.05463) by Yuanzhi Li, S\u00e9bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar and Yin Tat Lee.\n\n### Summary\n\nIn Phi-1 and Phi-1.5 papers, the authors showed how important the quality of the data is in training relative to the model size.\nThey selected high quality \"textbook\" data alongside with synthetically generated data for training their small sized Transformer\nbased model Phi-1 with 1.3B parameters. Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.\nThey follow the same strategy for Phi-1.5 and created another 1.3B parameter model with performance on natural language tasks comparable\nto models 5x larger, and surpassing most non-frontier LLMs. Phi-1.5 exhibits many of the traits of much larger LLMs such as the ability\nto \u201cthink step by step\u201d or perform some rudimentary in-context learning.\nWith these two experiments the authors successfully showed the huge impact of quality of training data when training machine learning models.\n\nThe abstract from the Phi-1 paper is the following:\n\n*We introduce phi-1, a new large language model for code, with significantly smaller size than\ncompeting models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on\n8 A100s, using a selection of \u201ctextbook quality\u201d data from the web (6B tokens) and synthetically\ngenerated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains\npass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent\nproperties compared to phi-1-base, our model before our finetuning stage on a dataset of coding\nexercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as\nphi-1 that still achieves 45% on HumanEval.*\n\nThe abstract from the Phi-1.5 paper is the following:\n\n*We continue the investigation into the power of smaller Transformer-based language models as\ninitiated by TinyStories \u2013 a 10 million parameter model that can produce coherent English \u2013 and\nthe follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close\nto the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to\ngenerate \u201ctextbook quality\u201d data as a way to enhance the learning process compared to traditional\nweb data."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e9cf2ca3-5513-459d-a455-1bf9c10be388
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e9cf2ca3-5513-459d-a455-1bf9c10be388\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f9ff5f3-b9c7-48e0-a4a0-046ca8500944\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7085f72a-ee7b-43e9-8331-4facb0ebd895\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1470e3b6fb408a41c406796d2477668d1d9e81f1b301bab0297c8d0ddc1cac30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45aaf0d8-c7d1-4a3d-9554-dd4122c33cf3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc38f22066d503c6187ba9898e841dd2c7173ed4fa013fdfe4680c2fb4924e86\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1251, \"end_char_idx\": 4372, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "ref_doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "chroma:document": "Phi-1.5 exhibits many of the traits of much larger LLMs such as the ability\nto \u201cthink step by step\u201d or perform some rudimentary in-context learning.\nWith these two experiments the authors successfully showed the huge impact of quality of training data when training machine learning models.\n\nThe abstract from the Phi-1 paper is the following:\n\n*We introduce phi-1, a new large language model for code, with significantly smaller size than\ncompeting models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on\n8 A100s, using a selection of \u201ctextbook quality\u201d data from the web (6B tokens) and synthetically\ngenerated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains\npass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent\nproperties compared to phi-1-base, our model before our finetuning stage on a dataset of coding\nexercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as\nphi-1 that still achieves 45% on HumanEval.*\n\nThe abstract from the Phi-1.5 paper is the following:\n\n*We continue the investigation into the power of smaller Transformer-based language models as\ninitiated by TinyStories \u2013 a 10 million parameter model that can produce coherent English \u2013 and\nthe follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close\nto the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to\ngenerate \u201ctextbook quality\u201d data as a way to enhance the learning process compared to traditional\nweb data. We follow the \u201cTextbooks Are All You Need\u201d approach, focusing this time on common\nsense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5,\nwith performance on natural language tasks comparable to models 5x larger, and surpassing most\nnon-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic\ncoding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good \u2013such\nas the ability to \u201cthink step by step\u201d or perform some rudimentary in-context learning\u2013 and bad,\nincluding hallucinations and the potential for toxic and biased generations \u2013encouragingly though, we\nare seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to\npromote further research on these urgent topics.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\n\nThe original code for Phi-1, Phi-1.5 and Phi-2 can be found [here](https://huggingface.co/microsoft/phi-1), [here](https://huggingface.co/microsoft/phi-1_5) and [here](https://huggingface.co/microsoft/phi-2), respectively.\n\n## Usage tips\n\n- This model is quite similar to `Llama` with the main difference in [`PhiDecoderLayer`], where they used [`PhiAttention`] and [`PhiMLP`] layers in parallel configuration.\n- The tokenizer used for this model is identical to the [`CodeGenTokenizer`].\n\n## How to use Phi-2\n\n<Tip warning={true}>\n\nPhi-2 has been integrated in the development version (4.37.0.dev) of `transformers`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 45aaf0d8-c7d1-4a3d-9554-dd4122c33cf3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"45aaf0d8-c7d1-4a3d-9554-dd4122c33cf3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f9ff5f3-b9c7-48e0-a4a0-046ca8500944\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9cf2ca3-5513-459d-a455-1bf9c10be388\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9d3a747800bd86c394bb91ee0c2490916715fa67c9078aa25226d1e51043a359\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"24331bc6-5d32-4298-a212-460114a8f445\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"638d20c204657d5239234b3165a3b06de54a47d483c94110b44c8a4dbbdffdb7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2864, \"end_char_idx\": 5975, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "ref_doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "chroma:document": "We follow the \u201cTextbooks Are All You Need\u201d approach, focusing this time on common\nsense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5,\nwith performance on natural language tasks comparable to models 5x larger, and surpassing most\nnon-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic\ncoding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good \u2013such\nas the ability to \u201cthink step by step\u201d or perform some rudimentary in-context learning\u2013 and bad,\nincluding hallucinations and the potential for toxic and biased generations \u2013encouragingly though, we\nare seeing improvement on that front thanks to the absence of web data. We open-source phi-1.5 to\npromote further research on these urgent topics.*\n\nThis model was contributed by [Susnato Dhar](https://huggingface.co/susnato).\n\nThe original code for Phi-1, Phi-1.5 and Phi-2 can be found [here](https://huggingface.co/microsoft/phi-1), [here](https://huggingface.co/microsoft/phi-1_5) and [here](https://huggingface.co/microsoft/phi-2), respectively.\n\n## Usage tips\n\n- This model is quite similar to `Llama` with the main difference in [`PhiDecoderLayer`], where they used [`PhiAttention`] and [`PhiMLP`] layers in parallel configuration.\n- The tokenizer used for this model is identical to the [`CodeGenTokenizer`].\n\n## How to use Phi-2\n\n<Tip warning={true}>\n\nPhi-2 has been integrated in the development version (4.37.0.dev) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/phi-2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\")\n\n>>> inputs = tokenizer('Can you help me write a formal email to a potential business partner proposing a joint venture?', return_tensors=\"pt\", return_attention_mask=False)\n\n>>> outputs = model.generate(**inputs, max_length=30)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\nCan you help me write a formal email to a potential business partner proposing a joint venture?\nInput: Company A: ABC Inc.\nCompany B\n```\n\n### Example :\n\n```python\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> # use the model to generate new tokens."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 24331bc6-5d32-4298-a212-460114a8f445
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"24331bc6-5d32-4298-a212-460114a8f445\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f9ff5f3-b9c7-48e0-a4a0-046ca8500944\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45aaf0d8-c7d1-4a3d-9554-dd4122c33cf3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a3c747c1c1e52e24e3b95b010205a24795048b982b06b3081a85f9021692999f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a4c2da21-c1ad-4f27-8d53-b261e3c00bf4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7b1844cd0723baaef3ce814dcfa12aaf633198619363a9910fb7bb4dd8ba8548\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4373, \"end_char_idx\": 7378, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "ref_doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "chroma:document": "Until the official version is released through `pip`, ensure that you are doing one of the following:\n\n* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.\n\n* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.\n\n</Tip>\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"microsoft/phi-2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-2\")\n\n>>> inputs = tokenizer('Can you help me write a formal email to a potential business partner proposing a joint venture?', return_tensors=\"pt\", return_attention_mask=False)\n\n>>> outputs = model.generate(**inputs, max_length=30)\n>>> text = tokenizer.batch_decode(outputs)[0]\n>>> print(text)\nCan you help me write a formal email to a potential business partner proposing a joint venture?\nInput: Company A: ABC Inc.\nCompany B\n```\n\n### Example :\n\n```python\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> # use the model to generate new tokens.\n>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)\n\n>>> tokenizer.batch_decode(generated_output)[0]\n'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'\n```\n\n## Combining Phi and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer and push the model and tokens to the GPU.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\")  # doctest: +SKIP\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n>>> # use the model to generate new tokens."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a4c2da21-c1ad-4f27-8d53-b261e3c00bf4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/phi", "title": "Phi", "tokens": 2257, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a4c2da21-c1ad-4f27-8d53-b261e3c00bf4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1f9ff5f3-b9c7-48e0-a4a0-046ca8500944\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f083b6893d1b95ff39f7a60bf1e880fd94e2d4cccc313e4b02560c5dcf2cfd72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"24331bc6-5d32-4298-a212-460114a8f445\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/phi\", \"title\": \"Phi\", \"tokens\": 2257, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2206a9704f5f80d5508d4f98fc78ffa23fb6646974598d0d9ebf06a341b4a1ee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5797, \"end_char_idx\": 8444, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "ref_doc_id": "1f9ff5f3-b9c7-48e0-a4a0-046ca8500944", "chroma:document": ">>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\")\n\n>>> # use the model to generate new tokens.\n>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)\n\n>>> tokenizer.batch_decode(generated_output)[0]\n'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'\n```\n\n## Combining Phi and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import PhiForCausalLM, AutoTokenizer\n\n>>> # define the model and tokenizer and push the model and tokens to the GPU.\n>>> model = PhiForCausalLM.from_pretrained(\"microsoft/phi-1_5\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\").to(\"cuda\")  # doctest: +SKIP\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/phi-1_5\")\n\n>>> # feel free to change the prompt to your liking.\n>>> prompt = \"If I were an AI that had just achieved\"\n\n>>> # apply the tokenizer.\n>>> tokens = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n>>> # use the model to generate new tokens.\n>>> generated_output = model.generate(**tokens, use_cache=True, max_new_tokens=10)  # doctest: +SKIP\n\n>>> tokenizer.batch_decode(generated_output)[0]  # doctest: +SKIP\n'If I were an AI that had just achieved a breakthrough in machine learning, I would be thrilled'\n```\n\n### Expected speedups\n\nBelow is an expected speedup diagram that compares pure inference time between the native implementation in transformers using `microsoft/phi-1` checkpoint and the Flash Attention 2 version of the model using a sequence length of 2048.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/phi_1_speedup_plot.jpg\">\n</div>\n\n## PhiConfig\n\n[[autodoc]] PhiConfig\n\n<frameworkcontent>\n<pt>\n\n## PhiModel\n\n[[autodoc]] PhiModel\n    - forward\n\n## PhiForCausalLM\n\n[[autodoc]] PhiForCausalLM\n    - forward\n    - generate\n\n## PhiForSequenceClassification\n\n[[autodoc]] PhiForSequenceClassification\n    - forward\n\n## PhiForTokenClassification\n\n[[autodoc]] PhiForTokenClassification\n    - forward\n\n</pt>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 7d7369ec-494a-467f-85ee-1ce55c0135d0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"7d7369ec-494a-467f-85ee-1ce55c0135d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d8ff5763-d294-4552-86c3-8b07832e65e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ca3936f-123a-4a9c-8700-6fdf87ec356e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b480bec7f264a3e1595308ceeaf69c432724d85b3d574536faab7df3c82178d7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2618, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "ref_doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "chroma:document": "# Auto Classes\n\nIn many cases, the architecture you want to use can be guessed from the name or the path of the pretrained model you\nare supplying to the `from_pretrained()` method. AutoClasses are here to do this job for you so that you\nautomatically retrieve the relevant model given the name/path to the pretrained weights/config/vocabulary.\n\nInstantiating one of [`AutoConfig`], [`AutoModel`], and\n[`AutoTokenizer`] will directly create a class of the relevant architecture. For instance\n\n\n```python\nmodel = AutoModel.from_pretrained(\"google-bert/bert-base-cased\")\n```\n\nwill create a model that is an instance of [`BertModel`].\n\nThere is one class of `AutoModel` for each task, and for each backend (PyTorch, TensorFlow, or Flax).\n\n## Extending the Auto Classes\n\nEach of the auto classes has a method to be extended with your custom classes. For instance, if you have defined a\ncustom class of model `NewModel`, make sure you have a `NewModelConfig` then you can add those to the auto\nclasses like this:\n\n```python\nfrom transformers import AutoConfig, AutoModel\n\nAutoConfig.register(\"new-model\", NewModelConfig)\nAutoModel.register(NewModelConfig, NewModel)\n```\n\nYou will then be able to use the auto classes like you would usually do!\n\n<Tip warning={true}>\n\nIf your `NewModelConfig` is a subclass of [`~transformers.PretrainedConfig`], make sure its\n`model_type` attribute is set to the same key you use when registering the config (here `\"new-model\"`).\n\nLikewise, if your `NewModel` is a subclass of [`PreTrainedModel`], make sure its\n`config_class` attribute is set to the same class you use when registering the model (here\n`NewModelConfig`).\n\n</Tip>\n\n## AutoConfig\n\n[[autodoc]] AutoConfig\n\n## AutoTokenizer\n\n[[autodoc]] AutoTokenizer\n\n## AutoFeatureExtractor\n\n[[autodoc]] AutoFeatureExtractor\n\n## AutoImageProcessor\n\n[[autodoc]] AutoImageProcessor\n\n## AutoProcessor\n\n[[autodoc]] AutoProcessor\n\n## Generic model classes\n\nThe following auto classes are available for instantiating a base model class without a specific head.\n\n### AutoModel\n\n[[autodoc]] AutoModel\n\n### TFAutoModel\n\n[[autodoc]] TFAutoModel\n\n### FlaxAutoModel\n\n[[autodoc]] FlaxAutoModel\n\n## Generic pretraining classes\n\nThe following auto classes are available for instantiating a model with a pretraining head.\n\n### AutoModelForPreTraining\n\n[[autodoc]] AutoModelForPreTraining\n\n### TFAutoModelForPreTraining\n\n[[autodoc]] TFAutoModelForPreTraining\n\n### FlaxAutoModelForPreTraining\n\n[[autodoc]] FlaxAutoModelForPreTraining\n\n## Natural Language Processing\n\nThe following auto classes are available for the following natural language processing tasks."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4ca3936f-123a-4a9c-8700-6fdf87ec356e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4ca3936f-123a-4a9c-8700-6fdf87ec356e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d8ff5763-d294-4552-86c3-8b07832e65e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d7369ec-494a-467f-85ee-1ce55c0135d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fbeaee0c0abee71615d5efa921523274b29cc6c27eae97f74a94dbe25d995b75\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6adb9fdd-7379-446e-b2ad-dc557f686e3f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30ae3c78bd44c2ef7cfe2313accaa334224d87f03ecaadd1f7793697758e7a77\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1240, \"end_char_idx\": 4875, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "ref_doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "chroma:document": "<Tip warning={true}>\n\nIf your `NewModelConfig` is a subclass of [`~transformers.PretrainedConfig`], make sure its\n`model_type` attribute is set to the same key you use when registering the config (here `\"new-model\"`).\n\nLikewise, if your `NewModel` is a subclass of [`PreTrainedModel`], make sure its\n`config_class` attribute is set to the same class you use when registering the model (here\n`NewModelConfig`).\n\n</Tip>\n\n## AutoConfig\n\n[[autodoc]] AutoConfig\n\n## AutoTokenizer\n\n[[autodoc]] AutoTokenizer\n\n## AutoFeatureExtractor\n\n[[autodoc]] AutoFeatureExtractor\n\n## AutoImageProcessor\n\n[[autodoc]] AutoImageProcessor\n\n## AutoProcessor\n\n[[autodoc]] AutoProcessor\n\n## Generic model classes\n\nThe following auto classes are available for instantiating a base model class without a specific head.\n\n### AutoModel\n\n[[autodoc]] AutoModel\n\n### TFAutoModel\n\n[[autodoc]] TFAutoModel\n\n### FlaxAutoModel\n\n[[autodoc]] FlaxAutoModel\n\n## Generic pretraining classes\n\nThe following auto classes are available for instantiating a model with a pretraining head.\n\n### AutoModelForPreTraining\n\n[[autodoc]] AutoModelForPreTraining\n\n### TFAutoModelForPreTraining\n\n[[autodoc]] TFAutoModelForPreTraining\n\n### FlaxAutoModelForPreTraining\n\n[[autodoc]] FlaxAutoModelForPreTraining\n\n## Natural Language Processing\n\nThe following auto classes are available for the following natural language processing tasks.\n\n### AutoModelForCausalLM\n\n[[autodoc]] AutoModelForCausalLM\n\n### TFAutoModelForCausalLM\n\n[[autodoc]] TFAutoModelForCausalLM\n\n### FlaxAutoModelForCausalLM\n\n[[autodoc]] FlaxAutoModelForCausalLM\n\n### AutoModelForMaskedLM\n\n[[autodoc]] AutoModelForMaskedLM\n\n### TFAutoModelForMaskedLM\n\n[[autodoc]] TFAutoModelForMaskedLM\n\n### FlaxAutoModelForMaskedLM\n\n[[autodoc]] FlaxAutoModelForMaskedLM\n\n### AutoModelForMaskGeneration\n\n[[autodoc]] AutoModelForMaskGeneration\n\n### TFAutoModelForMaskGeneration\n\n[[autodoc]] TFAutoModelForMaskGeneration\n\n### AutoModelForSeq2SeqLM\n\n[[autodoc]] AutoModelForSeq2SeqLM\n\n### TFAutoModelForSeq2SeqLM\n\n[[autodoc]] TFAutoModelForSeq2SeqLM\n\n### FlaxAutoModelForSeq2SeqLM\n\n[[autodoc]] FlaxAutoModelForSeq2SeqLM\n\n### AutoModelForSequenceClassification\n\n[[autodoc]] AutoModelForSequenceClassification\n\n### TFAutoModelForSequenceClassification\n\n[[autodoc]] TFAutoModelForSequenceClassification\n\n### FlaxAutoModelForSequenceClassification\n\n[[autodoc]] FlaxAutoModelForSequenceClassification\n\n### AutoModelForMultipleChoice\n\n[[autodoc]] AutoModelForMultipleChoice\n\n### TFAutoModelForMultipleChoice\n\n[[autodoc]] TFAutoModelForMultipleChoice\n\n### FlaxAutoModelForMultipleChoice\n\n[[autodoc]] FlaxAutoModelForMultipleChoice\n\n### AutoModelForNextSentencePrediction\n\n[[autodoc]] AutoModelForNextSentencePrediction\n\n### TFAutoModelForNextSentencePrediction\n\n[[autodoc]] TFAutoModelForNextSentencePrediction\n\n### FlaxAutoModelForNextSentencePrediction\n\n[[autodoc]] FlaxAutoModelForNextSentencePrediction\n\n### AutoModelForTokenClassification\n\n[[autodoc]] AutoModelForTokenClassification\n\n### TFAutoModelForTokenClassification\n\n[[autodoc]] TFAutoModelForTokenClassification\n\n### FlaxAutoModelForTokenClassification\n\n[[autodoc]] FlaxAutoModelForTokenClassification\n\n### AutoModelForQuestionAnswering\n\n[[autodoc]] AutoModelForQuestionAnswering\n\n### TFAutoModelForQuestionAnswering\n\n[[autodoc]] TFAutoModelForQuestionAnswering\n\n### FlaxAutoModelForQuestionAnswering\n\n[[autodoc]] FlaxAutoModelForQuestionAnswering\n\n### AutoModelForTextEncoding\n\n[[autodoc]] AutoModelForTextEncoding\n\n### TFAutoModelForTextEncoding\n\n[[autodoc]] TFAutoModelForTextEncoding\n\n## Computer vision\n\nThe following auto classes are available for the following computer vision tasks."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6adb9fdd-7379-446e-b2ad-dc557f686e3f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6adb9fdd-7379-446e-b2ad-dc557f686e3f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d8ff5763-d294-4552-86c3-8b07832e65e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ca3936f-123a-4a9c-8700-6fdf87ec356e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5dfbc35aa8ffb60d9be5e806b8ff4cf18434f991b9198c3cef4c07b72fca79c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af017b8c-3fb5-4824-a0fd-28a9a790e49b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"483b737e38141b09087203cb31518ffd892e9da53950bfdd4658aacf7413e188\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4877, \"end_char_idx\": 7318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "ref_doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "chroma:document": "### AutoModelForDepthEstimation\n\n[[autodoc]] AutoModelForDepthEstimation\n\n### AutoModelForImageClassification\n\n[[autodoc]] AutoModelForImageClassification\n\n### TFAutoModelForImageClassification\n\n[[autodoc]] TFAutoModelForImageClassification\n\n### FlaxAutoModelForImageClassification\n\n[[autodoc]] FlaxAutoModelForImageClassification\n\n### AutoModelForVideoClassification\n\n[[autodoc]] AutoModelForVideoClassification\n\n### AutoModelForKeypointDetection\n\n[[autodoc]] AutoModelForKeypointDetection\n\n### AutoModelForMaskedImageModeling\n\n[[autodoc]] AutoModelForMaskedImageModeling\n\n### TFAutoModelForMaskedImageModeling\n\n[[autodoc]] TFAutoModelForMaskedImageModeling\n\n### AutoModelForObjectDetection\n\n[[autodoc]] AutoModelForObjectDetection\n\n### AutoModelForImageSegmentation\n\n[[autodoc]] AutoModelForImageSegmentation\n\n### AutoModelForImageToImage\n\n[[autodoc]] AutoModelForImageToImage\n\n### AutoModelForSemanticSegmentation\n\n[[autodoc]] AutoModelForSemanticSegmentation\n\n### TFAutoModelForSemanticSegmentation\n\n[[autodoc]] TFAutoModelForSemanticSegmentation\n\n### AutoModelForInstanceSegmentation\n\n[[autodoc]] AutoModelForInstanceSegmentation\n\n### AutoModelForUniversalSegmentation\n\n[[autodoc]] AutoModelForUniversalSegmentation\n\n### AutoModelForZeroShotImageClassification\n\n[[autodoc]] AutoModelForZeroShotImageClassification\n\n### TFAutoModelForZeroShotImageClassification\n\n[[autodoc]] TFAutoModelForZeroShotImageClassification\n\n### AutoModelForZeroShotObjectDetection\n\n[[autodoc]] AutoModelForZeroShotObjectDetection\n\n## Audio\n\nThe following auto classes are available for the following audio tasks.\n\n### AutoModelForAudioClassification\n\n[[autodoc]] AutoModelForAudioClassification\n\n### AutoModelForAudioFrameClassification\n\n[[autodoc]] TFAutoModelForAudioClassification\n\n### TFAutoModelForAudioFrameClassification\n\n[[autodoc]] AutoModelForAudioFrameClassification\n\n### AutoModelForCTC\n\n[[autodoc]] AutoModelForCTC\n\n### AutoModelForSpeechSeq2Seq\n\n[[autodoc]] AutoModelForSpeechSeq2Seq\n\n### TFAutoModelForSpeechSeq2Seq\n\n[[autodoc]] TFAutoModelForSpeechSeq2Seq\n\n### FlaxAutoModelForSpeechSeq2Seq\n\n[[autodoc]] FlaxAutoModelForSpeechSeq2Seq\n\n### AutoModelForAudioXVector\n\n[[autodoc]] AutoModelForAudioXVector\n\n### AutoModelForTextToSpectrogram\n\n[[autodoc]] AutoModelForTextToSpectrogram\n\n### AutoModelForTextToWaveform\n\n[[autodoc]] AutoModelForTextToWaveform\n\n## Multimodal\n\nThe following auto classes are available for the following multimodal tasks."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: af017b8c-3fb5-4824-a0fd-28a9a790e49b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/auto", "title": "Auto Classes", "tokens": 2236, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"af017b8c-3fb5-4824-a0fd-28a9a790e49b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d8ff5763-d294-4552-86c3-8b07832e65e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d8f0f619a1e8679c754dcf0599555b80506f888e81147fc2280d915e250d55e6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6adb9fdd-7379-446e-b2ad-dc557f686e3f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/auto\", \"title\": \"Auto Classes\", \"tokens\": 2236, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"2a424f3a6ad8688527df8c59d0496f74dc902cc29fee1a2f3d4e4f9617a2b22e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6472, \"end_char_idx\": 7984, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "ref_doc_id": "d8ff5763-d294-4552-86c3-8b07832e65e2", "chroma:document": "### AutoModelForAudioClassification\n\n[[autodoc]] AutoModelForAudioClassification\n\n### AutoModelForAudioFrameClassification\n\n[[autodoc]] TFAutoModelForAudioClassification\n\n### TFAutoModelForAudioFrameClassification\n\n[[autodoc]] AutoModelForAudioFrameClassification\n\n### AutoModelForCTC\n\n[[autodoc]] AutoModelForCTC\n\n### AutoModelForSpeechSeq2Seq\n\n[[autodoc]] AutoModelForSpeechSeq2Seq\n\n### TFAutoModelForSpeechSeq2Seq\n\n[[autodoc]] TFAutoModelForSpeechSeq2Seq\n\n### FlaxAutoModelForSpeechSeq2Seq\n\n[[autodoc]] FlaxAutoModelForSpeechSeq2Seq\n\n### AutoModelForAudioXVector\n\n[[autodoc]] AutoModelForAudioXVector\n\n### AutoModelForTextToSpectrogram\n\n[[autodoc]] AutoModelForTextToSpectrogram\n\n### AutoModelForTextToWaveform\n\n[[autodoc]] AutoModelForTextToWaveform\n\n## Multimodal\n\nThe following auto classes are available for the following multimodal tasks.\n\n### AutoModelForTableQuestionAnswering\n\n[[autodoc]] AutoModelForTableQuestionAnswering\n\n### TFAutoModelForTableQuestionAnswering\n\n[[autodoc]] TFAutoModelForTableQuestionAnswering\n\n### AutoModelForDocumentQuestionAnswering\n\n[[autodoc]] AutoModelForDocumentQuestionAnswering\n\n### TFAutoModelForDocumentQuestionAnswering\n\n[[autodoc]] TFAutoModelForDocumentQuestionAnswering\n\n### AutoModelForVisualQuestionAnswering\n\n[[autodoc]] AutoModelForVisualQuestionAnswering\n\n### AutoModelForVision2Seq\n\n[[autodoc]] AutoModelForVision2Seq\n\n### TFAutoModelForVision2Seq\n\n[[autodoc]] TFAutoModelForVision2Seq\n\n### FlaxAutoModelForVision2Seq\n\n[[autodoc]] FlaxAutoModelForVision2Seq"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 13d7373a-b508-46c5-85ed-8262c5c6768c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/instructblip", "title": "InstructBLIP", "tokens": 835, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"13d7373a-b508-46c5-85ed-8262c5c6768c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/instructblip\", \"title\": \"InstructBLIP\", \"tokens\": 835, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5b56e6d9-bf04-4f88-82e7-52a00f3feb91\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/instructblip\", \"title\": \"InstructBLIP\", \"tokens\": 835, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8edb96b9117f51e50618d63f1fbc41cc4f982e5820e5158038031b15ec22f7d8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5b56e6d9-bf04-4f88-82e7-52a00f3feb91", "doc_id": "5b56e6d9-bf04-4f88-82e7-52a00f3feb91", "ref_doc_id": "5b56e6d9-bf04-4f88-82e7-52a00f3feb91", "chroma:document": "# InstructBLIP\n\n## Overview\n\nThe InstructBLIP model was proposed in [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://arxiv.org/abs/2305.06500) by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.\nInstructBLIP leverages the [BLIP-2](blip2) architecture for visual instruction tuning.\n\nThe abstract from the paper is the following:\n\n*General-purpose language models that can solve various language-domain tasks have emerged driven by the pre-training and instruction-tuning pipeline. However, building general-purpose vision-language models is challenging due to the increased task discrepancy introduced by the additional visual input. Although vision-language pre-training has been widely studied, vision-language instruction tuning remains relatively less explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pre-trained BLIP-2 models. We gather a wide variety of 26 publicly available datasets, transform them into instruction tuning format and categorize them into two clusters for held-in instruction tuning and held-out zero-shot evaluation. Additionally, we introduce instruction-aware visual feature extraction, a crucial method that enables the model to extract informative features tailored to the given instruction. The resulting InstructBLIP models achieve state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and the larger Flamingo. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA IMG). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/instructblip_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> InstructBLIP architecture. Taken from the <a href=\"https://arxiv.org/abs/2305.06500\">original paper.</a> </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/salesforce/LAVIS/tree/main/projects/instructblip).\n\n## Usage tips\n\nInstructBLIP uses the same architecture as [BLIP-2](blip2) with a tiny but important difference: it also feeds the text prompt (instruction) to the Q-Former.\n\n## InstructBlipConfig\n\n[[autodoc]] InstructBlipConfig\n    - from_vision_qformer_text_configs\n\n## InstructBlipVisionConfig\n\n[[autodoc]] InstructBlipVisionConfig\n\n## InstructBlipQFormerConfig\n\n[[autodoc]] InstructBlipQFormerConfig\n\n## InstructBlipProcessor\n\n[[autodoc]] InstructBlipProcessor\n\n\n## InstructBlipVisionModel\n\n[[autodoc]] InstructBlipVisionModel\n    - forward\n\n## InstructBlipQFormerModel\n\n[[autodoc]] InstructBlipQFormerModel\n    - forward\n\n## InstructBlipForConditionalGeneration\n\n[[autodoc]] InstructBlipForConditionalGeneration\n    - forward\n    - generate"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a973fa20-1579-4bbc-b282-9d32f60f3733
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_audio", "title": "Qwen2Audio", "tokens": 2398, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a973fa20-1579-4bbc-b282-9d32f60f3733\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85984a95-c82b-4fd6-80be-2238eabc1e39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cfc98d6e63eed43028f52b37f7e948c07c901ff9c378099a5bfa2d128e6e57c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ed73f7c6-3180-4b27-a87c-7fdae362305a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f82f4e99a41ba1c20743d0699b1c0d301c8aa8876bc811ef6c79bbd9a9692cc9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2910, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "ref_doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "chroma:document": "# Qwen2Audio\n\n## Overview\n\nThe Qwen2-Audio is the new model series of large audio-language models from the Qwen team. Qwen2-Audio is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. We introduce two distinct audio interaction modes:\n\n* voice chat: users can freely engage in voice interactions with Qwen2-Audio without text input\n* audio analysis: users could provide audio and text instructions for analysis during the interaction\n\nIt was proposed in [Qwen2-Audio Technical Report](https://arxiv.org/abs/2407.10759) by Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, Jingren Zhou. \n\nThe abstract from the paper is the following:\n\n*We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community. *\n\n\n## Usage tips\n\n`Qwen2-Audio-7B` and `Qwen2-Audio-7B-Instruct` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)\n\nIn the following, we demonstrate how to use `Qwen2-Audio-7B-Instruct` for the inference, supporting both voice chat and audio analysis modes. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ed73f7c6-3180-4b27-a87c-7fdae362305a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_audio", "title": "Qwen2Audio", "tokens": 2398, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ed73f7c6-3180-4b27-a87c-7fdae362305a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85984a95-c82b-4fd6-80be-2238eabc1e39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cfc98d6e63eed43028f52b37f7e948c07c901ff9c378099a5bfa2d128e6e57c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a973fa20-1579-4bbc-b282-9d32f60f3733\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e1a50a9b7d52ba292695b8089e4698b4737d0b6dab9aa8b396354e4f5d3acd11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4f3c5915-854a-4d38-801c-45c8b2768b40\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e5f4d05879187e0120b722c07f7d51ebb0be91c9cf5309f21eef4f1ba67ba41a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2509, \"end_char_idx\": 5515, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "ref_doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "chroma:document": "## Usage tips\n\n`Qwen2-Audio-7B` and `Qwen2-Audio-7B-Instruct` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)\n\nIn the following, we demonstrate how to use `Qwen2-Audio-7B-Instruct` for the inference, supporting both voice chat and audio analysis modes. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.\n\n### Voice Chat Inference\nIn the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input:\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import Qwen2AudioForConditionalGeneration, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"Qwen/Qwen2-Audio-7B-Instruct\")\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen/Qwen2-Audio-7B-Instruct\", device_map=\"auto\")\n\nconversation = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/guess_age_gender.wav\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"Yes, the speaker is female and in her twenties.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/translate_to_chinese.wav\"},\n    ]},\n]\ntext = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)\naudios = []\nfor message in conversation:\n    if isinstance(message[\"content\"], list):\n        for ele in message[\"content\"]:\n            if ele[\"type\"] == \"audio\":\n                audios.append(librosa.load(\n                    BytesIO(urlopen(ele['audio_url']).read()), \n                    sr=processor.feature_extractor.sampling_rate)[0]\n                )\n\ninputs = processor(text=text, audios=audios, return_tensors=\"pt\", padding=True)\ninputs.input_ids = inputs.input_ids.to(\"cuda\")\n\ngenerate_ids = model.generate(**inputs, max_length=256)\ngenerate_ids = generate_ids[:, inputs.input_ids.size(1):]\n\nresponse = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n```\n\n### Audio Analysis Inference\nIn the audio analysis, users could provide both audio and text instructions for analysis:\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import Qwen2AudioForConditionalGeneration, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"Qwen/Qwen2-Audio-7B-Instruct\")\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen/Qwen2-Audio-7B-Instruct\", device_map=\"auto\")\n\nconversation = [\n    {'role': 'system', 'content': 'You are a helpful assistant.'}, \n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3\"},\n        {\"type\": \"text\", \"text\": \"What's that sound?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"It is the sound of glass shattering."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4f3c5915-854a-4d38-801c-45c8b2768b40
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_audio", "title": "Qwen2Audio", "tokens": 2398, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4f3c5915-854a-4d38-801c-45c8b2768b40\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85984a95-c82b-4fd6-80be-2238eabc1e39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cfc98d6e63eed43028f52b37f7e948c07c901ff9c378099a5bfa2d128e6e57c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed73f7c6-3180-4b27-a87c-7fdae362305a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a9cc39872dbd5dc6f81aacf881f95c67a51f805e20d1f616ef52b81a428ccb81\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9d4c3ed4-4190-4a9e-8d60-b1c14d9341be\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"162025b9664e975408932e5f1dea29eb75cddf1624f3db6defa9366805754053\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5203, \"end_char_idx\": 8054, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "ref_doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "chroma:document": "'}, \n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3\"},\n        {\"type\": \"text\", \"text\": \"What's that sound?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"It is the sound of glass shattering.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"text\", \"text\": \"What can you do when you hear that?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"Stay alert and cautious, and check if anyone is hurt or if there is any damage to property.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac\"},\n        {\"type\": \"text\", \"text\": \"What does the person say?\"},\n    ]},\n]\ntext = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)\naudios = []\nfor message in conversation:\n    if isinstance(message[\"content\"], list):\n        for ele in message[\"content\"]:\n            if ele[\"type\"] == \"audio\":\n                audios.append(\n                    librosa.load(\n                        BytesIO(urlopen(ele['audio_url']).read()), \n                        sr=processor.feature_extractor.sampling_rate)[0]\n                )\n\ninputs = processor(text=text, audios=audios, return_tensors=\"pt\", padding=True)\ninputs.input_ids = inputs.input_ids.to(\"cuda\")\n\ngenerate_ids = model.generate(**inputs, max_length=256)\ngenerate_ids = generate_ids[:, inputs.input_ids.size(1):]\n\nresponse = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n```\n\n### Batch Inference\nWe also support batch inference:\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import Qwen2AudioForConditionalGeneration, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"Qwen/Qwen2-Audio-7B-Instruct\")\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen/Qwen2-Audio-7B-Instruct\", device_map=\"auto\")\n\nconversation1 = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3\"},\n        {\"type\": \"text\", \"text\": \"What's that sound?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"It is the sound of glass shattering.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/f2641_0_throatclearing.wav\"},\n        {\"type\": \"text\", \"text\": \"What can you hear?\"},\n    ]}\n]\n\nconversation2 = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac\"},\n        {\"type\": \"text\", \"text\": \"What does the person say?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9d4c3ed4-4190-4a9e-8d60-b1c14d9341be
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_audio", "title": "Qwen2Audio", "tokens": 2398, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9d4c3ed4-4190-4a9e-8d60-b1c14d9341be\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"85984a95-c82b-4fd6-80be-2238eabc1e39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"3cfc98d6e63eed43028f52b37f7e948c07c901ff9c378099a5bfa2d128e6e57c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4f3c5915-854a-4d38-801c-45c8b2768b40\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_audio\", \"title\": \"Qwen2Audio\", \"tokens\": 2398, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e808854216f9583c113f42d46fe34cb9fe9c71a98475e41ae419d1ad846ffd9c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7486, \"end_char_idx\": 9385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "ref_doc_id": "85984a95-c82b-4fd6-80be-2238eabc1e39", "chroma:document": "\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"It is the sound of glass shattering.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/f2641_0_throatclearing.wav\"},\n        {\"type\": \"text\", \"text\": \"What can you hear?\"},\n    ]}\n]\n\nconversation2 = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac\"},\n        {\"type\": \"text\", \"text\": \"What does the person say?\"},\n    ]},\n]\n\nconversations = [conversation1, conversation2]\n\ntext = [processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) for conversation in conversations]\n\naudios = []\nfor conversation in conversations:\n    for message in conversation:\n        if isinstance(message[\"content\"], list):\n            for ele in message[\"content\"]:\n                if ele[\"type\"] == \"audio\":\n                    audios.append(\n                        librosa.load(\n                            BytesIO(urlopen(ele['audio_url']).read()), \n                            sr=processor.feature_extractor.sampling_rate)[0]\n                    )\n\ninputs = processor(text=text, audios=audios, return_tensors=\"pt\", padding=True)\ninputs['input_ids'] = inputs['input_ids'].to(\"cuda\")\ninputs.input_ids = inputs.input_ids.to(\"cuda\")\n\ngenerate_ids = model.generate(**inputs, max_length=256)\ngenerate_ids = generate_ids[:, inputs.input_ids.size(1):]\n\nresponse = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\n\n## Qwen2AudioConfig\n\n[[autodoc]] Qwen2AudioConfig\n\n## Qwen2AudioConfig\n\n[[autodoc]] Qwen2AudioEncoderConfig\n\n## Qwen2AudioProcessor\n\n[[autodoc]] Qwen2AudioProcessor\n\n## Qwen2AudioForConditionalGeneration\n\n[[autodoc]] Qwen2AudioForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9e94fbe8-77b4-4fa2-ad7a-ee9efcbd7c13
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9e94fbe8-77b4-4fa2-ad7a-ee9efcbd7c13\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b3c8fd70-f3a5-476a-be65-448d67ab77d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c2cc174a1d7856c89dd0b4ef11ae76fe6155bb64ae49e86b5519453ac5ab3700\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d55d60ae-361c-4fa5-8a38-6ec01bab2e2d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52d5ef87898cc8306b9bf41e59d2a3a99819c8436dd87d1095668ccc9a0ad455\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1959, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b3c8fd70-f3a5-476a-be65-448d67ab77d3", "doc_id": "b3c8fd70-f3a5-476a-be65-448d67ab77d3", "ref_doc_id": "b3c8fd70-f3a5-476a-be65-448d67ab77d3", "chroma:document": "# MPNet\n\n## Overview\n\nThe MPNet model was proposed in [MPNet: Masked and Permuted Pre-training for Language Understanding](https://arxiv.org/abs/2004.09297) by Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu.\n\nMPNet adopts a novel pre-training method, named masked and permuted language modeling, to inherit the advantages of\nmasked language modeling and permuted language modeling for natural language understanding.\n\nThe abstract from the paper is the following:\n\n*BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models.\nSince BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for\npre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and\nthus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel\npre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the\ndependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position\ninformation as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in\nXLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of\ndown-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large\nmargin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g.,\nBERT, XLNet, RoBERTa) under the same model setting.*\n\nThe original code can be found [here](https://github.com/microsoft/MPNet).\n\n## Usage tips\n\nMPNet doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just \nseparate your segments with the separation token `tokenizer.sep_token` (or `[sep]`)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d55d60ae-361c-4fa5-8a38-6ec01bab2e2d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mpnet", "title": "MPNet", "tokens": 1049, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d55d60ae-361c-4fa5-8a38-6ec01bab2e2d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b3c8fd70-f3a5-476a-be65-448d67ab77d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c2cc174a1d7856c89dd0b4ef11ae76fe6155bb64ae49e86b5519453ac5ab3700\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9e94fbe8-77b4-4fa2-ad7a-ee9efcbd7c13\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mpnet\", \"title\": \"MPNet\", \"tokens\": 1049, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e2ceeedf1db052add18d8a07114e18bc5e29d1097773428f446c2c50b77c17fb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 217, \"end_char_idx\": 3568, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b3c8fd70-f3a5-476a-be65-448d67ab77d3", "doc_id": "b3c8fd70-f3a5-476a-be65-448d67ab77d3", "ref_doc_id": "b3c8fd70-f3a5-476a-be65-448d67ab77d3", "chroma:document": "MPNet adopts a novel pre-training method, named masked and permuted language modeling, to inherit the advantages of\nmasked language modeling and permuted language modeling for natural language understanding.\n\nThe abstract from the paper is the following:\n\n*BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models.\nSince BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for\npre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and\nthus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel\npre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the\ndependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position\ninformation as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in\nXLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of\ndown-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large\nmargin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g.,\nBERT, XLNet, RoBERTa) under the same model setting.*\n\nThe original code can be found [here](https://github.com/microsoft/MPNet).\n\n## Usage tips\n\nMPNet doesn't have `token_type_ids`, you don't need to indicate which token belongs to which segment. Just \nseparate your segments with the separation token `tokenizer.sep_token` (or `[sep]`).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## MPNetConfig\n\n[[autodoc]] MPNetConfig\n\n## MPNetTokenizer\n\n[[autodoc]] MPNetTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## MPNetTokenizerFast\n\n[[autodoc]] MPNetTokenizerFast\n\n<frameworkcontent>\n<pt>\n\n## MPNetModel\n\n[[autodoc]] MPNetModel\n    - forward\n\n## MPNetForMaskedLM\n\n[[autodoc]] MPNetForMaskedLM\n    - forward\n\n## MPNetForSequenceClassification\n\n[[autodoc]] MPNetForSequenceClassification\n    - forward\n\n## MPNetForMultipleChoice\n\n[[autodoc]] MPNetForMultipleChoice\n    - forward\n\n## MPNetForTokenClassification\n\n[[autodoc]] MPNetForTokenClassification\n    - forward\n\n## MPNetForQuestionAnswering\n\n[[autodoc]] MPNetForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFMPNetModel\n\n[[autodoc]] TFMPNetModel\n    - call\n\n## TFMPNetForMaskedLM\n\n[[autodoc]] TFMPNetForMaskedLM\n    - call\n\n## TFMPNetForSequenceClassification\n\n[[autodoc]] TFMPNetForSequenceClassification\n    - call\n\n## TFMPNetForMultipleChoice\n\n[[autodoc]] TFMPNetForMultipleChoice\n    - call\n\n## TFMPNetForTokenClassification\n\n[[autodoc]] TFMPNetForTokenClassification\n    - call\n\n## TFMPNetForQuestionAnswering\n\n[[autodoc]] TFMPNetForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 48885842-5869-494e-a24f-98dea91eaaa6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"48885842-5869-494e-a24f-98dea91eaaa6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc9cab1-467e-45d8-b18e-416f8d226e23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7400bbb015bd9b3b1bc661b1c0729a4d1b519c73160f31ba722b6b7fc56c4bf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6cc40142-5fbc-4f41-845a-cab78e915476\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4c629fbaf95d990a1d8d6b7a2426a232158392b387882bcc01f1a99fcf3c04d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3244, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc9cab1-467e-45d8-b18e-416f8d226e23", "doc_id": "0bc9cab1-467e-45d8-b18e-416f8d226e23", "ref_doc_id": "0bc9cab1-467e-45d8-b18e-416f8d226e23", "chroma:document": "# ConvNeXT\n\n## Overview\n\nThe ConvNeXT model was proposed in [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545) by Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie.\nConvNeXT is a pure convolutional model (ConvNet), inspired by the design of Vision Transformers, that claims to outperform them.\n\nThe abstract from the paper is the following:\n\n*The \"Roaring 20s\" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model.\nA vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers\n(e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide\nvariety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive\nbiases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually \"modernize\" a standard ResNet toward the design\nof a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models\ndubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy\nand outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ConvNeXT architecture. Taken from the <a href=\"https://arxiv.org/abs/2201.03545\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [ariG23498](https://github.com/ariG23498),\n[gante](https://github.com/gante), and [sayakpaul](https://github.com/sayakpaul) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ConvNeXT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ConvNextForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6cc40142-5fbc-4f41-845a-cab78e915476
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/convnext", "title": "ConvNeXT", "tokens": 1043, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6cc40142-5fbc-4f41-845a-cab78e915476\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0bc9cab1-467e-45d8-b18e-416f8d226e23\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b7400bbb015bd9b3b1bc661b1c0729a4d1b519c73160f31ba722b6b7fc56c4bf\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"48885842-5869-494e-a24f-98dea91eaaa6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/convnext\", \"title\": \"ConvNeXT\", \"tokens\": 1043, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"dd241be7ac9d4fc423b264baa8a6a15228b6242accc681a5b3de030b85f3f78c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1528, \"end_char_idx\": 3808, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0bc9cab1-467e-45d8-b18e-416f8d226e23", "doc_id": "0bc9cab1-467e-45d8-b18e-416f8d226e23", "ref_doc_id": "0bc9cab1-467e-45d8-b18e-416f8d226e23", "chroma:document": "Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy\nand outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> ConvNeXT architecture. Taken from the <a href=\"https://arxiv.org/abs/2201.03545\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [ariG23498](https://github.com/ariG23498),\n[gante](https://github.com/gante), and [sayakpaul](https://github.com/sayakpaul) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/ConvNeXt).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ConvNeXT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`ConvNextForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ConvNextConfig\n\n[[autodoc]] ConvNextConfig\n\n## ConvNextFeatureExtractor\n\n[[autodoc]] ConvNextFeatureExtractor\n\n## ConvNextImageProcessor\n\n[[autodoc]] ConvNextImageProcessor\n    - preprocess\n\n<frameworkcontent>\n<pt>\n\n## ConvNextModel\n\n[[autodoc]] ConvNextModel\n    - forward\n\n## ConvNextForImageClassification\n\n[[autodoc]] ConvNextForImageClassification\n    - forward\n\n</pt>\n<tf>\n\n## TFConvNextModel\n\n[[autodoc]] TFConvNextModel\n    - call\n\n## TFConvNextForImageClassification\n\n[[autodoc]] TFConvNextForImageClassification\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 67d39eee-c68f-485f-95bd-eeb4cd90781c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"67d39eee-c68f-485f-95bd-eeb4cd90781c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9346c3a3-154d-4067-b4a9-ca72fc6339d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12b53caa-4a50-4d76-b3ea-f70c5f64a9a9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6ef2972ab154af51afa8fa35ef8672b0eedf9815107eb22c4e604abb57e6c1e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3177, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "ref_doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "chroma:document": "# SegFormer\n\n## Overview\n\nThe SegFormer model was proposed in [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://arxiv.org/abs/2105.15203) by Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping\nLuo. The model consists of a hierarchical Transformer encoder and a lightweight all-MLP decode head to achieve great\nresults on image segmentation benchmarks such as ADE20K and Cityscapes.\n\nThe abstract from the paper is the following:\n\n*We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with\nlightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel\nhierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding,\nthereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution\ndiffers from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from\ndifferent layers, and thus combining both local attention and global attention to render powerful representations. We\nshow that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our\napproach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance\nand efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters,\nbeing 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on\nCityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.*\n\nThe figure below illustrates the architecture of SegFormer. Taken from the [original paper](https://arxiv.org/abs/2105.15203).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/segformer_architecture.png\"/>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version\nof the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/NVlabs/SegFormer).\n\n## Usage tips\n\n- SegFormer consists of a hierarchical Transformer encoder, and a lightweight all-MLP decoder head.\n  [`SegformerModel`] is the hierarchical Transformer encoder (which in the paper is also referred to\n  as Mix Transformer or MiT). [`SegformerForSemanticSegmentation`] adds the all-MLP decoder head on\n  top to perform semantic segmentation of images. In addition, there's\n  [`SegformerForImageClassification`] which can be used to - you guessed it - classify images. The\n  authors of SegFormer first pre-trained the Transformer encoder on ImageNet-1k to classify images. Next, they throw\n  away the classification head, and replace it by the all-MLP decode head. Next, they fine-tune the model altogether on\n  ADE20K, Cityscapes and COCO-stuff, which are important benchmarks for semantic segmentation. All checkpoints can be\n  found on the [hub](https://huggingface.co/models?other=segformer)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 12b53caa-4a50-4d76-b3ea-f70c5f64a9a9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"12b53caa-4a50-4d76-b3ea-f70c5f64a9a9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9346c3a3-154d-4067-b4a9-ca72fc6339d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"67d39eee-c68f-485f-95bd-eeb4cd90781c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1eaeccc05b7340451e3d3aa42c788d5004060bef70ddd63bd012072643dfa167\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6803121-7e40-4db3-9d4a-d457919b64c0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f336006770b1f40293791d20ff21088930ffa729eb1d39035ab249b25f68baaa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1620, \"end_char_idx\": 4785, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "ref_doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "chroma:document": "Our best model, SegFormer-B5, achieves 84.0% mIoU on\nCityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C.*\n\nThe figure below illustrates the architecture of SegFormer. Taken from the [original paper](https://arxiv.org/abs/2105.15203).\n\n<img width=\"600\" src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/segformer_architecture.png\"/>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The TensorFlow version\nof the model was contributed by [sayakpaul](https://huggingface.co/sayakpaul). The original code can be found [here](https://github.com/NVlabs/SegFormer).\n\n## Usage tips\n\n- SegFormer consists of a hierarchical Transformer encoder, and a lightweight all-MLP decoder head.\n  [`SegformerModel`] is the hierarchical Transformer encoder (which in the paper is also referred to\n  as Mix Transformer or MiT). [`SegformerForSemanticSegmentation`] adds the all-MLP decoder head on\n  top to perform semantic segmentation of images. In addition, there's\n  [`SegformerForImageClassification`] which can be used to - you guessed it - classify images. The\n  authors of SegFormer first pre-trained the Transformer encoder on ImageNet-1k to classify images. Next, they throw\n  away the classification head, and replace it by the all-MLP decode head. Next, they fine-tune the model altogether on\n  ADE20K, Cityscapes and COCO-stuff, which are important benchmarks for semantic segmentation. All checkpoints can be\n  found on the [hub](https://huggingface.co/models?other=segformer).\n- The quickest way to get started with SegFormer is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer) (which showcase both inference and\n  fine-tuning on custom data). One can also check out the [blog post](https://huggingface.co/blog/fine-tune-segformer) introducing SegFormer and illustrating how it can be fine-tuned on custom data.\n- TensorFlow users should refer to [this repository](https://github.com/deep-diver/segformer-tf-transformers) that shows off-the-shelf inference and fine-tuning.\n- One can also check out [this interactive demo on Hugging Face Spaces](https://huggingface.co/spaces/chansung/segformer-tf-transformers)\n  to try out a SegFormer model on custom images.\n- SegFormer works on any input size, as it pads the input to be divisible by `config.patch_sizes`.\n- One can use [`SegformerImageProcessor`] to prepare images and corresponding segmentation maps\n  for the model. Note that this image processor is fairly basic and does not include all data augmentations used in\n  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most\n  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,\n  such as 512x512 or 640x640, after which they are normalized.\n- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with\n  `do_reduce_labels` set to `True` or `False`."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: f6803121-7e40-4db3-9d4a-d457919b64c0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"f6803121-7e40-4db3-9d4a-d457919b64c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9346c3a3-154d-4067-b4a9-ca72fc6339d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12b53caa-4a50-4d76-b3ea-f70c5f64a9a9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4b9311c0177fdf39ba8718abe34d67cb967dded220cc7692bc95672340eb6772\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"965a421e-26e7-4f34-a817-907859dfda07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46576fe9aea188e6b532d6b6f1a1dae1a076ba0c16b7649ffbaa395e43fef3c8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3178, \"end_char_idx\": 5659, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "ref_doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "chroma:document": "- The quickest way to get started with SegFormer is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer) (which showcase both inference and\n  fine-tuning on custom data). One can also check out the [blog post](https://huggingface.co/blog/fine-tune-segformer) introducing SegFormer and illustrating how it can be fine-tuned on custom data.\n- TensorFlow users should refer to [this repository](https://github.com/deep-diver/segformer-tf-transformers) that shows off-the-shelf inference and fine-tuning.\n- One can also check out [this interactive demo on Hugging Face Spaces](https://huggingface.co/spaces/chansung/segformer-tf-transformers)\n  to try out a SegFormer model on custom images.\n- SegFormer works on any input size, as it pads the input to be divisible by `config.patch_sizes`.\n- One can use [`SegformerImageProcessor`] to prepare images and corresponding segmentation maps\n  for the model. Note that this image processor is fairly basic and does not include all data augmentations used in\n  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most\n  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,\n  such as 512x512 or 640x640, after which they are normalized.\n- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with\n  `do_reduce_labels` set to `True` or `False`. In some datasets (like ADE20k), the 0 index is used in the annotated\n  segmentation maps for background. However, ADE20k doesn't include the \"background\" class in its 150 labels.\n  Therefore, `do_reduce_labels` is used to reduce all labels by 1, and to make sure no loss is computed for the\n  background class (i.e. it replaces 0 in the annotated maps by 255, which is the *ignore_index* of the loss function\n  used by [`SegformerForSemanticSegmentation`]). However, other datasets use the 0 index as\n  background class and include this class as part of all labels. In that case, `do_reduce_labels` should be set to\n  `False`, as loss should also be computed for the background class.\n- As most models, SegFormer comes in different sizes, the details of which can be found in the table below\n  (taken from Table 7 of the [original paper](https://arxiv.org/abs/2105.15203))."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 965a421e-26e7-4f34-a817-907859dfda07
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"965a421e-26e7-4f34-a817-907859dfda07\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9346c3a3-154d-4067-b4a9-ca72fc6339d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6803121-7e40-4db3-9d4a-d457919b64c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1a7fc2fd6d781aa5cc5dfb8b6999f5f30821a29999c478895a302090acc9a4d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"239820a2-154d-461d-b5a6-39a6685aaf2c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f30e53434db5327c93966eeb77321e6568c73e00f85384acb87057ed9d0547c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4137, \"end_char_idx\": 6782, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "ref_doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "chroma:document": "Note that this image processor is fairly basic and does not include all data augmentations used in\n  the original paper. The original preprocessing pipelines (for the ADE20k dataset for instance) can be found [here](https://github.com/NVlabs/SegFormer/blob/master/local_configs/_base_/datasets/ade20k_repeat.py). The most\n  important preprocessing step is that images and segmentation maps are randomly cropped and padded to the same size,\n  such as 512x512 or 640x640, after which they are normalized.\n- One additional thing to keep in mind is that one can initialize [`SegformerImageProcessor`] with\n  `do_reduce_labels` set to `True` or `False`. In some datasets (like ADE20k), the 0 index is used in the annotated\n  segmentation maps for background. However, ADE20k doesn't include the \"background\" class in its 150 labels.\n  Therefore, `do_reduce_labels` is used to reduce all labels by 1, and to make sure no loss is computed for the\n  background class (i.e. it replaces 0 in the annotated maps by 255, which is the *ignore_index* of the loss function\n  used by [`SegformerForSemanticSegmentation`]). However, other datasets use the 0 index as\n  background class and include this class as part of all labels. In that case, `do_reduce_labels` should be set to\n  `False`, as loss should also be computed for the background class.\n- As most models, SegFormer comes in different sizes, the details of which can be found in the table below\n  (taken from Table 7 of the [original paper](https://arxiv.org/abs/2105.15203)).\n\n| **Model variant** | **Depths**    | **Hidden sizes**    | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :---------------------: | :------------: | :-------------------: |\n| MiT-b0            | [2, 2, 2, 2]  | [32, 64, 160, 256]  | 256                     | 3.7            | 70.5                  |\n| MiT-b1            | [2, 2, 2, 2]  | [64, 128, 320, 512] | 256                     | 14.0           | 78.7                  |\n| MiT-b2            | [3, 4, 6, 3]  | [64, 128, 320, 512] | 768                     | 25.4           | 81.6                  |\n| MiT-b3            | [3, 4, 18, 3] | [64, 128, 320, 512] | 768                     | 45.2           | 83.1                  |\n| MiT-b4            | [3, 8, 27, 3] | [64, 128, 320, 512] | 768                     | 62.6           | 83.6                  |\n| MiT-b5            | [3, 6, 40, 3] | [64, 128, 320, 512] | 768                     | 82.0           | 83.8                  |\n\nNote that MiT in the above table refers to the Mix Transformer encoder backbone introduced in SegFormer."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 239820a2-154d-461d-b5a6-39a6685aaf2c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"239820a2-154d-461d-b5a6-39a6685aaf2c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9346c3a3-154d-4067-b4a9-ca72fc6339d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"965a421e-26e7-4f34-a817-907859dfda07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0e6fbe4f9d9b0b0785321383926f805a26b103243b97686fc6e1dcdaa61e33c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c9709a79-eb8e-4a18-a3ef-9f8a3c9a7762\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc0f3945eafb535d4a7deb531a0249224ee1918e045a2b964b3483e6a5c1beb8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5661, \"end_char_idx\": 8418, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "ref_doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "chroma:document": "| **Model variant** | **Depths**    | **Hidden sizes**    | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n| :---------------: | ------------- | ------------------- | :---------------------: | :------------: | :-------------------: |\n| MiT-b0            | [2, 2, 2, 2]  | [32, 64, 160, 256]  | 256                     | 3.7            | 70.5                  |\n| MiT-b1            | [2, 2, 2, 2]  | [64, 128, 320, 512] | 256                     | 14.0           | 78.7                  |\n| MiT-b2            | [3, 4, 6, 3]  | [64, 128, 320, 512] | 768                     | 25.4           | 81.6                  |\n| MiT-b3            | [3, 4, 18, 3] | [64, 128, 320, 512] | 768                     | 45.2           | 83.1                  |\n| MiT-b4            | [3, 8, 27, 3] | [64, 128, 320, 512] | 768                     | 62.6           | 83.6                  |\n| MiT-b5            | [3, 6, 40, 3] | [64, 128, 320, 512] | 768                     | 82.0           | 83.8                  |\n\nNote that MiT in the above table refers to the Mix Transformer encoder backbone introduced in SegFormer. For\nSegFormer's results on the segmentation datasets like ADE20k, refer to the [paper](https://arxiv.org/abs/2105.15203).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SegFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SegformerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- [Image classification task guide](../tasks/image_classification)\n\nSemantic segmentation:\n\n- [`SegformerForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation).\n- A blog on fine-tuning SegFormer on a custom dataset can be found [here](https://huggingface.co/blog/fine-tune-segformer).\n- More demo notebooks on SegFormer (both inference + fine-tuning on a custom dataset) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).\n- [`TFSegformerForSemanticSegmentation`] is supported by this [example notebook](https://github.com/huggingface/notebooks/blob/main/examples/semantic_segmentation-tf.ipynb).\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: c9709a79-eb8e-4a18-a3ef-9f8a3c9a7762
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/segformer", "title": "SegFormer", "tokens": 2515, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"c9709a79-eb8e-4a18-a3ef-9f8a3c9a7762\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9346c3a3-154d-4067-b4a9-ca72fc6339d7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"fa13ad5cf1bef8af89feaa3c34fad14e14c9ad6528d7f801e6dcd9530f70bee1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"239820a2-154d-461d-b5a6-39a6685aaf2c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/segformer\", \"title\": \"SegFormer\", \"tokens\": 2515, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a4d784ef7c4d4ac1745d3e42a69370a904e8919575e1fe98a40d8eb65efeb8f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6783, \"end_char_idx\": 9431, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "ref_doc_id": "9346c3a3-154d-4067-b4a9-ca72fc6339d7", "chroma:document": "For\nSegFormer's results on the segmentation datasets like ADE20k, refer to the [paper](https://arxiv.org/abs/2105.15203).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with SegFormer.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- [`SegformerForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- [Image classification task guide](../tasks/image_classification)\n\nSemantic segmentation:\n\n- [`SegformerForSemanticSegmentation`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation).\n- A blog on fine-tuning SegFormer on a custom dataset can be found [here](https://huggingface.co/blog/fine-tune-segformer).\n- More demo notebooks on SegFormer (both inference + fine-tuning on a custom dataset) can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/SegFormer).\n- [`TFSegformerForSemanticSegmentation`] is supported by this [example notebook](https://github.com/huggingface/notebooks/blob/main/examples/semantic_segmentation-tf.ipynb).\n- [Semantic segmentation task guide](../tasks/semantic_segmentation)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## SegformerConfig\n\n[[autodoc]] SegformerConfig\n\n## SegformerFeatureExtractor\n\n[[autodoc]] SegformerFeatureExtractor\n    - __call__\n    - post_process_semantic_segmentation\n\n## SegformerImageProcessor\n\n[[autodoc]] SegformerImageProcessor\n    - preprocess\n    - post_process_semantic_segmentation\n\n<frameworkcontent>\n<pt>\n\n## SegformerModel\n\n[[autodoc]] SegformerModel\n    - forward\n\n## SegformerDecodeHead\n\n[[autodoc]] SegformerDecodeHead\n    - forward\n\n## SegformerForImageClassification\n\n[[autodoc]] SegformerForImageClassification\n    - forward\n\n## SegformerForSemanticSegmentation\n\n[[autodoc]] SegformerForSemanticSegmentation\n    - forward\n\n</pt>\n<tf>\n\n## TFSegformerDecodeHead\n\n[[autodoc]] TFSegformerDecodeHead\n    - call\n\n## TFSegformerModel\n\n[[autodoc]] TFSegformerModel\n    - call\n\n## TFSegformerForImageClassification\n\n[[autodoc]] TFSegformerForImageClassification\n    - call\n\n## TFSegformerForSemanticSegmentation\n\n[[autodoc]] TFSegformerForSemanticSegmentation\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ea4c50aa-eae5-4451-a85e-1c3123aa3eec
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xls_r", "title": "XLS-R", "tokens": 705, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ea4c50aa-eae5-4451-a85e-1c3123aa3eec\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xls_r\", \"title\": \"XLS-R\", \"tokens\": 705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8e940e9f-46b4-409a-a937-d17e5a5ef1c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xls_r\", \"title\": \"XLS-R\", \"tokens\": 705, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b6a3b7c5d6f8afedad8afb749f0acaf2bd55b9787da7c18949f944f4bb293ad9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2170, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8e940e9f-46b4-409a-a937-d17e5a5ef1c6", "doc_id": "8e940e9f-46b4-409a-a937-d17e5a5ef1c6", "ref_doc_id": "8e940e9f-46b4-409a-a937-d17e5a5ef1c6", "chroma:document": "# XLS-R\n\n## Overview\n\nThe XLS-R model was proposed in [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://arxiv.org/abs/2111.09296) by Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman\nGoyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli.\n\nThe abstract from the paper is the following:\n\n*This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0.\nWe train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128\nlanguages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range\nof tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation\nbenchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into\nEnglish. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as\nVoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107\nlanguage identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform\nEnglish-only pretraining when translating English speech into other languages, a setting which favors monolingual\npretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.*\n\nRelevant checkpoints can be found under https://huggingface.co/models?other=xls_r.\n\nThe original code can be found [here](https://github.com/pytorch/fairseq/tree/master/fairseq/models/wav2vec).\n\n## Usage tips\n\n- XLS-R is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n- XLS-R model was trained using connectionist temporal classification (CTC) so the model output has to be decoded using\n  [`Wav2Vec2CTCTokenizer`].\n\n<Tip>\n\nXLS-R's architecture is based on the Wav2Vec2 model, refer to [Wav2Vec2's documentation page](wav2vec2) for API reference.\n\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0fcd7412-b816-430e-9df8-0fadd8af2801
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/xlm-v", "title": "XLM-V", "tokens": 706, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0fcd7412-b816-430e-9df8-0fadd8af2801\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-v\", \"title\": \"XLM-V\", \"tokens\": 706, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f8222fac-db77-4a34-bdf3-bdf6e02c494e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/xlm-v\", \"title\": \"XLM-V\", \"tokens\": 706, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"c72cdce69276b484c8bd8986ad08c800c44a050a05119c20f640a6f1d7368bb4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f8222fac-db77-4a34-bdf3-bdf6e02c494e", "doc_id": "f8222fac-db77-4a34-bdf3-bdf6e02c494e", "ref_doc_id": "f8222fac-db77-4a34-bdf3-bdf6e02c494e", "chroma:document": "# XLM-V\n\n## Overview\n\nXLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R).\nIt was introduced in the [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://arxiv.org/abs/2301.10472)\npaper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa.\n\nFrom the abstract of the XLM-V paper:\n\n*Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.\nAs these models have increased in parameter count and depth, vocabulary size has remained largely unchanged.\nThis vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R.\nIn this paper, we introduce a new approach for scaling to very large multilingual vocabularies by\nde-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity\nto achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically\nmore semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V,\na multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we\ntested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and\nnamed entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).*\n\nThis model was contributed by [stefan-it](https://huggingface.co/stefan-it), including detailed experiments with XLM-V on downstream tasks.\nThe experiments repository can be found [here](https://github.com/stefan-it/xlm-v-experiments).\n\n## Usage tips\n\n- XLM-V is compatible with the XLM-RoBERTa model architecture, only model weights from [`fairseq`](https://github.com/facebookresearch/fairseq)\n  library had to be converted.\n- The `XLMTokenizer` implementation is used to load the vocab and performs tokenization.\n\nA XLM-V (base size) model is available under the [`facebook/xlm-v-base`](https://huggingface.co/facebook/xlm-v-base) identifier.\n\n<Tip>\n\nXLM-V architecture is the same as XLM-RoBERTa, refer to [XLM-RoBERTa documentation](xlm-roberta) for API reference, and examples.\n</Tip>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: dcfa7896-0396-446d-9335-1d496400dab7
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"dcfa7896-0396-446d-9335-1d496400dab7\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e2f35695-534e-4f41-8ed9-ee199c5c0aef\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b8fd130c73c9d2f7f023e2ceaac967ac46fe27725a3288b49c1ab37d1308c6f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e73346cf-1a71-465e-9f54-2e2d92685def\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c8e97a119d7ccfd449e3f9e46958d008acf9378dc4ebd41c8edab5cf1b3063ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1894, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e2f35695-534e-4f41-8ed9-ee199c5c0aef", "doc_id": "e2f35695-534e-4f41-8ed9-ee199c5c0aef", "ref_doc_id": "e2f35695-534e-4f41-8ed9-ee199c5c0aef", "chroma:document": "# RemBERT\n\n## Overview\n\nThe RemBERT model was proposed in [Rethinking Embedding Coupling in Pre-trained Language Models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault F\u00e9vry, Henry Tsai, Melvin Johnson, Sebastian Ruder.\n\nThe abstract from the paper is the following:\n\n*We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art\npre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to\nsignificantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By\nreallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on\nstandard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that\nallocating additional capacity to the output embedding provides benefits to the model that persist through the\nfine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger\noutput embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage\nTransformer representations to be more general and more transferable to other tasks and languages. Harnessing these\nfindings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the\nnumber of parameters at the fine-tuning stage.*\n\n## Usage tips\n\nFor fine-tuning, RemBERT can be thought of as a bigger version of mBERT with an ALBERT-like factorization of the\nembedding layer. The embeddings are not tied in pre-training, in contrast with BERT, which enables smaller input\nembeddings (preserved during fine-tuning) and bigger output embeddings (discarded at fine-tuning). The tokenizer is\nalso similar to the Albert one rather than the BERT one."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e73346cf-1a71-465e-9f54-2e2d92685def
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/rembert", "title": "RemBERT", "tokens": 1117, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e73346cf-1a71-465e-9f54-2e2d92685def\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e2f35695-534e-4f41-8ed9-ee199c5c0aef\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b8fd130c73c9d2f7f023e2ceaac967ac46fe27725a3288b49c1ab37d1308c6f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dcfa7896-0396-446d-9335-1d496400dab7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/rembert\", \"title\": \"RemBERT\", \"tokens\": 1117, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a89c5ae4a57e10ec6fcaac34b18a33bee80086f2713a2d6bc3b0cbd1afef8ed9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3904, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e2f35695-534e-4f41-8ed9-ee199c5c0aef", "doc_id": "e2f35695-534e-4f41-8ed9-ee199c5c0aef", "ref_doc_id": "e2f35695-534e-4f41-8ed9-ee199c5c0aef", "chroma:document": "# RemBERT\n\n## Overview\n\nThe RemBERT model was proposed in [Rethinking Embedding Coupling in Pre-trained Language Models](https://arxiv.org/abs/2010.12821) by Hyung Won Chung, Thibault F\u00e9vry, Henry Tsai, Melvin Johnson, Sebastian Ruder.\n\nThe abstract from the paper is the following:\n\n*We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art\npre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to\nsignificantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By\nreallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on\nstandard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that\nallocating additional capacity to the output embedding provides benefits to the model that persist through the\nfine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger\noutput embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage\nTransformer representations to be more general and more transferable to other tasks and languages. Harnessing these\nfindings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the\nnumber of parameters at the fine-tuning stage.*\n\n## Usage tips\n\nFor fine-tuning, RemBERT can be thought of as a bigger version of mBERT with an ALBERT-like factorization of the\nembedding layer. The embeddings are not tied in pre-training, in contrast with BERT, which enables smaller input\nembeddings (preserved during fine-tuning) and bigger output embeddings (discarded at fine-tuning). The tokenizer is\nalso similar to the Albert one rather than the BERT one.\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Causal language modeling task guide](../tasks/language_modeling)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## RemBertConfig\n\n[[autodoc]] RemBertConfig\n\n## RemBertTokenizer\n\n[[autodoc]] RemBertTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## RemBertTokenizerFast\n\n[[autodoc]] RemBertTokenizerFast\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n<frameworkcontent>\n<pt>\n\n## RemBertModel\n\n[[autodoc]] RemBertModel\n    - forward\n\n## RemBertForCausalLM\n\n[[autodoc]] RemBertForCausalLM\n    - forward\n\n## RemBertForMaskedLM\n\n[[autodoc]] RemBertForMaskedLM\n    - forward\n\n## RemBertForSequenceClassification\n\n[[autodoc]] RemBertForSequenceClassification\n    - forward\n\n## RemBertForMultipleChoice\n\n[[autodoc]] RemBertForMultipleChoice\n    - forward\n\n## RemBertForTokenClassification\n\n[[autodoc]] RemBertForTokenClassification\n    - forward\n\n## RemBertForQuestionAnswering\n\n[[autodoc]] RemBertForQuestionAnswering\n    - forward\n\n</pt>\n<tf>\n\n## TFRemBertModel\n\n[[autodoc]] TFRemBertModel\n    - call\n\n## TFRemBertForMaskedLM\n\n[[autodoc]] TFRemBertForMaskedLM\n    - call\n\n## TFRemBertForCausalLM\n\n[[autodoc]] TFRemBertForCausalLM\n    - call\n\n## TFRemBertForSequenceClassification\n\n[[autodoc]] TFRemBertForSequenceClassification\n    - call\n\n## TFRemBertForMultipleChoice\n\n[[autodoc]] TFRemBertForMultipleChoice\n    - call\n\n## TFRemBertForTokenClassification\n\n[[autodoc]] TFRemBertForTokenClassification\n    - call\n\n## TFRemBertForQuestionAnswering\n\n[[autodoc]] TFRemBertForQuestionAnswering\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e70a8026-fc80-4425-aceb-5474c59ea341
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/qwen2_moe", "title": "Qwen2MoE", "tokens": 941, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e70a8026-fc80-4425-aceb-5474c59ea341\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_moe\", \"title\": \"Qwen2MoE\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3799bf18-516b-4bb6-936d-c1053ba59f7e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/qwen2_moe\", \"title\": \"Qwen2MoE\", \"tokens\": 941, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e5ae81ea9610210ac4b74678145129ff55c1a67580f0f1aac667046df0cd8f76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2797, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3799bf18-516b-4bb6-936d-c1053ba59f7e", "doc_id": "3799bf18-516b-4bb6-936d-c1053ba59f7e", "ref_doc_id": "3799bf18-516b-4bb6-936d-c1053ba59f7e", "chroma:document": "# Qwen2MoE\n\n## Overview\n\nQwen2MoE is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen-72B, Qwen-1.8B, Qwen-VL, Qwen-Audio, etc.\n\n### Model Details\n\nQwen2MoE is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. Qwen2MoE has the following architectural choices:\n\n- Qwen2MoE is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.\n- Qwen2MoE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, `Qwen1.5-MoE-A2.7B` is upcycled from `Qwen-1.8B`. It has 14.3B parameters in total and 2.7B activated parameters during runtime, while it achieves comparable performance with `Qwen1.5-7B`, with only 25% of the training resources.\n\nFor more details refer to the [release blog post](https://qwenlm.github.io/blog/qwen-moe/).\n\n## Usage tips\n\n`Qwen1.5-MoE-A2.7B` and `Qwen1.5-MoE-A2.7B-Chat` can be found on the [Huggingface Hub](https://huggingface.co/Qwen)\n\nIn the following, we demonstrate how to use `Qwen1.5-MoE-A2.7B-Chat` for the inference. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen1.5-MoE-A2.7B-Chat\", device_map=\"auto\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen1.5-MoE-A2.7B-Chat\")\n\n>>> prompt = \"Give me a short introduction to large language model.\"\n\n>>> messages = [{\"role\": \"user\", \"content\": prompt}]\n\n>>> text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n\n>>> model_inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n\n>>> generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=True)\n\n>>> generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)]\n\n>>> response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\n## Qwen2MoeConfig\n\n[[autodoc]] Qwen2MoeConfig\n\n## Qwen2MoeModel\n\n[[autodoc]] Qwen2MoeModel\n    - forward\n\n## Qwen2MoeForCausalLM\n\n[[autodoc]] Qwen2MoeForCausalLM\n    - forward\n\n## Qwen2MoeForSequenceClassification\n\n[[autodoc]] Qwen2MoeForSequenceClassification\n    - forward\n\n## Qwen2MoeForTokenClassification\n\n[[autodoc]] Qwen2MoeForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a629bbd1-b67b-49bc-9e5b-642fcc019a69
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a629bbd1-b67b-49bc-9e5b-642fcc019a69\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6b8fd96c-a9c7-4685-a597-ed9202ad55e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd11802a-2cfa-42b3-8195-bfa79d59e87e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c545c881b354c5d2123665c77a39fa2246c915604548056bcb996a838e31c2ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3388, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "ref_doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "chroma:document": "# SeamlessM4T\n\n## Overview\n\nThe SeamlessM4T model was proposed in [SeamlessM4T \u2014 Massively Multilingual & Multimodal Machine Translation](https://dl.fbaipublicfiles.com/seamless/seamless_m4t_paper.pdf) by the Seamless Communication team from Meta AI.\n\nThis is the **version 1** release of the model. For the updated **version 2** release, refer to the [Seamless M4T v2 docs](https://huggingface.co/docs/transformers/main/model_doc/seamless_m4t_v2).\n\nSeamlessM4T is a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text.\n\nSeamlessM4T enables multiple tasks without relying on separate models:\n\n- Speech-to-speech translation (S2ST)\n- Speech-to-text translation (S2TT)\n- Text-to-speech translation (T2ST)\n- Text-to-text translation (T2TT)\n- Automatic speech recognition (ASR)\n\n[`SeamlessM4TModel`] can perform all the above tasks, but each task also has its own dedicated sub-model.\n\nThe abstract from the paper is the following:\n\n*What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication*\n\n## Usage\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4TModel\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n>>> model = SeamlessM4TModel.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cd11802a-2cfa-42b3-8195-bfa79d59e87e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cd11802a-2cfa-42b3-8195-bfa79d59e87e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6b8fd96c-a9c7-4685-a597-ed9202ad55e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a629bbd1-b67b-49bc-9e5b-642fcc019a69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bdd724ea97dcaeb9a9d5d05751c5ee132863d46e1d6db18d2d6bd536d86edb44\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"04e62cd8-de24-4859-b0da-2933bb9ae275\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f3fd281d257beb3f8ad6c171d794c992afe58c24c2b5508bb87f8d03d8c5c086\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1800, \"end_char_idx\": 5077, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "ref_doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "chroma:document": "To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication*\n\n## Usage\n\nFirst, load the processor and a checkpoint of the model:\n\n```python\n>>> from transformers import AutoProcessor, SeamlessM4TModel\n\n>>> processor = AutoProcessor.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n>>> model = SeamlessM4TModel.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nYou can seamlessly use this model on text or on audio, to generated either translated text or translated audio.\n\nHere is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English test as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```\n\n\n### Speech\n\n[`SeamlessM4TModel`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4TModel.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 04e62cd8-de24-4859-b0da-2933bb9ae275
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"04e62cd8-de24-4859-b0da-2933bb9ae275\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6b8fd96c-a9c7-4685-a597-ed9202ad55e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cd11802a-2cfa-42b3-8195-bfa79d59e87e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0886e82448a2f79aefe6bb08baf388af3dd2fabd9915c278bf29ceebbb47c686\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5328116f-c985-459b-aaa9-9b0b061cb0cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0baafe24a31e2b0559b9a0369a6c30203f6eef4fb92f3b5d0e655aed494f848b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3390, \"end_char_idx\": 6745, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "ref_doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "chroma:document": "Here is how to use the processor to process text and audio:\n\n```python\n>>> # let's load an audio sample from an Arabic speech corpus\n>>> from datasets import load_dataset\n>>> dataset = load_dataset(\"arabic_speech_corpus\", split=\"test\", streaming=True)\n>>> audio_sample = next(iter(dataset))[\"audio\"]\n\n>>> # now, process it\n>>> audio_inputs = processor(audios=audio_sample[\"array\"], return_tensors=\"pt\")\n\n>>> # now, process some English test as well\n>>> text_inputs = processor(text = \"Hello, my dog is cute\", src_lang=\"eng\", return_tensors=\"pt\")\n```\n\n\n### Speech\n\n[`SeamlessM4TModel`] can *seamlessly* generate text or speech with few or no changes. Let's target Russian voice translation:\n\n```python\n>>> audio_array_from_text = model.generate(**text_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n>>> audio_array_from_audio = model.generate(**audio_inputs, tgt_lang=\"rus\")[0].cpu().numpy().squeeze()\n```\n\nWith basically the same code, I've translated English text and Arabic speech to Russian speech samples.\n\n### Text\n\nSimilarly, you can generate translated text from audio files or from text with the same model. You only have to pass `generate_speech=False` to [`SeamlessM4TModel.generate`].\nThis time, let's translate to French.\n\n```python \n>>> # from audio\n>>> output_tokens = model.generate(**audio_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_audio = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n\n>>> # from text\n>>> output_tokens = model.generate(**text_inputs, tgt_lang=\"fra\", generate_speech=False)\n>>> translated_text_from_text = processor.decode(output_tokens[0].tolist()[0], skip_special_tokens=True)\n```\n\n### Tips\n\n\n#### 1. Use dedicated models\n\n[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.\nFor example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: \n\n```python\n>>> from transformers import SeamlessM4TForSpeechToSpeech\n>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nOr you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.\n\n```python\n>>> from transformers import SeamlessM4TForTextToText\n>>> model = SeamlessM4TForTextToText.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nFeel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !\n\n## Model architecture"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 5328116f-c985-459b-aaa9-9b0b061cb0cd
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"5328116f-c985-459b-aaa9-9b0b061cb0cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6b8fd96c-a9c7-4685-a597-ed9202ad55e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"04e62cd8-de24-4859-b0da-2933bb9ae275\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"04cd0b58a8c6e7420b2f7a58fac9c04580b657134a11a87f9972c15bad3499b4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e2dcc0af-e119-47e6-ab8d-0dd511d33803\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"31224a1abbd7ba94ccb28d25bb81b1aeacfa787a9a29eafa2a9f43532e08eae4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5080, \"end_char_idx\": 8138, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "ref_doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "chroma:document": "#### 1. Use dedicated models\n\n[`SeamlessM4TModel`] is transformers top level model to generate speech and text, but you can also use dedicated models that perform the task without additional components, thus reducing the memory footprint.\nFor example, you can replace the audio-to-audio generation snippet with the model dedicated to the S2ST task, the rest is exactly the same code: \n\n```python\n>>> from transformers import SeamlessM4TForSpeechToSpeech\n>>> model = SeamlessM4TForSpeechToSpeech.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nOr you can replace the text-to-text generation snippet with the model dedicated to the T2TT task, you only have to remove `generate_speech=False`.\n\n```python\n>>> from transformers import SeamlessM4TForTextToText\n>>> model = SeamlessM4TForTextToText.from_pretrained(\"facebook/hf-seamless-m4t-medium\")\n```\n\nFeel free to try out [`SeamlessM4TForSpeechToText`] and [`SeamlessM4TForTextToSpeech`] as well.\n\n#### 2. Change the speaker identity\n\nYou have the possibility to change the speaker used for speech synthesis with the `spkr_id` argument. Some `spkr_id` works better than other for some languages!\n\n#### 3. Change the generation strategy\n\nYou can use different [generation strategies](./generation_strategies) for speech and text generation, e.g `.generate(input_ids=input_ids, text_num_beams=4, speech_do_sample=True)` which will successively perform beam-search decoding on the text model, and multinomial sampling on the speech model.\n\n#### 4. Generate speech and text at the same time\n\nUse `return_intermediate_token_ids=True` with [`SeamlessM4TModel`] to return both speech and text !\n\n## Model architecture\n\n\nSeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text.\n\nEach modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.\n- These unit tokens are then passed through the final vocoder to produce the actual speech.\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## SeamlessM4TModel\n\n[[autodoc]] SeamlessM4TModel\n    - generate\n\n\n## SeamlessM4TForTextToSpeech\n\n[[autodoc]] SeamlessM4TForTextToSpeech\n    - generate\n\n\n## SeamlessM4TForSpeechToSpeech\n\n[[autodoc]] SeamlessM4TForSpeechToSpeech\n    - generate"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: e2dcc0af-e119-47e6-ab8d-0dd511d33803
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/seamless_m4t", "title": "SeamlessM4T", "tokens": 2285, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"e2dcc0af-e119-47e6-ab8d-0dd511d33803\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6b8fd96c-a9c7-4685-a597-ed9202ad55e2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"bd14ce182ae2788028072f22741058c7eeacc5898f9bcf2e24820c1e2a779104\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5328116f-c985-459b-aaa9-9b0b061cb0cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/seamless_m4t\", \"title\": \"SeamlessM4T\", \"tokens\": 2285, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9eddd8099b9a468e15b8f2880e09c4e48e91914d4e9dd81ae516d215df6507a0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6748, \"end_char_idx\": 9167, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "ref_doc_id": "6b8fd96c-a9c7-4685-a597-ed9202ad55e2", "chroma:document": "SeamlessM4T features a versatile architecture that smoothly handles the sequential generation of text and speech. This setup comprises two sequence-to-sequence (seq2seq) models. The first model translates the input modality into translated text, while the second model generates speech tokens, known as \"unit tokens,\" from the translated text.\n\nEach modality has its own dedicated encoder with a unique architecture. Additionally, for speech output, a vocoder inspired by the [HiFi-GAN](https://arxiv.org/abs/2010.05646) architecture is placed on top of the second seq2seq model.\n\nHere's how the generation process works:\n\n- Input text or speech is processed through its specific encoder.\n- A decoder creates text tokens in the desired language.\n- If speech generation is required, the second seq2seq model, following a standard encoder-decoder structure, generates unit tokens.\n- These unit tokens are then passed through the final vocoder to produce the actual speech.\n\n\nThis model was contributed by [ylacombe](https://huggingface.co/ylacombe). The original code can be found [here](https://github.com/facebookresearch/seamless_communication).\n\n## SeamlessM4TModel\n\n[[autodoc]] SeamlessM4TModel\n    - generate\n\n\n## SeamlessM4TForTextToSpeech\n\n[[autodoc]] SeamlessM4TForTextToSpeech\n    - generate\n\n\n## SeamlessM4TForSpeechToSpeech\n\n[[autodoc]] SeamlessM4TForSpeechToSpeech\n    - generate\n\n\n## SeamlessM4TForTextToText\n\n[[autodoc]] transformers.SeamlessM4TForTextToText\n    - forward\n    - generate\n\n## SeamlessM4TForSpeechToText\n\n[[autodoc]] transformers.SeamlessM4TForSpeechToText\n    - forward\n    - generate\n\n## SeamlessM4TConfig\n\n[[autodoc]] SeamlessM4TConfig\n\n\n## SeamlessM4TTokenizer\n\n[[autodoc]] SeamlessM4TTokenizer\n    - __call__\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n\n## SeamlessM4TTokenizerFast\n\n[[autodoc]] SeamlessM4TTokenizerFast\n    - __call__\n\n## SeamlessM4TFeatureExtractor\n\n[[autodoc]] SeamlessM4TFeatureExtractor\n    - __call__\n\n## SeamlessM4TProcessor\n\n[[autodoc]] SeamlessM4TProcessor\n    - __call__\n\n## SeamlessM4TCodeHifiGan\n\n[[autodoc]] SeamlessM4TCodeHifiGan\n\n\n## SeamlessM4THifiGan\n\n[[autodoc]] SeamlessM4THifiGan\n\n## SeamlessM4TTextToUnitModel\n\n[[autodoc]] SeamlessM4TTextToUnitModel\n\n## SeamlessM4TTextToUnitForConditionalGeneration\n\n[[autodoc]] SeamlessM4TTextToUnitForConditionalGeneration"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2dfc2567-ab93-42a7-a142-7ec6c135c881
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2dfc2567-ab93-42a7-a142-7ec6c135c881\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"72da8e79-38c2-403e-b0ee-e773bc3fd119\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"14e118e1-0e8a-4f93-bc77-3b9ff5621fa9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e20dafc9d6623ee78a309cc42aabc93d29c085b753dc87c9290905ac1c24940e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "ref_doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "chroma:document": "# ImageGPT\n\n## Overview\n\nThe ImageGPT model was proposed in [Generative Pretraining from Pixels](https://openai.com/blog/image-gpt) by Mark\nChen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever. ImageGPT (iGPT) is a GPT-2-like\nmodel trained to predict the next pixel value, allowing for both unconditional and conditional image generation.\n\nThe abstract from the paper is the following:\n\n*Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models\ncan learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels,\nwithout incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels,\nwe find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and\nlow-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide\nResNet, and 99.0% accuracy with full fine-tuning, matching the top supervised pre-trained models. We are also\ncompetitive with self-supervised benchmarks on ImageNet when substituting pixels for a VQVAE encoding, achieving 69.0%\ntop-1 accuracy on a linear probe of our features.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/imagegpt_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Summary of the approach. Taken from the [original paper](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf). </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr), based on [this issue](https://github.com/openai/image-gpt/issues/7). The original code can be found\n[here](https://github.com/openai/image-gpt).\n\n## Usage tips\n\n- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation\n  function is used (namely \"quick gelu\"), and the layer normalization layers don't mean center the inputs. ImageGPT\n  also doesn't have tied input- and output embeddings.\n- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence\n  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a\n  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors\n  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long\n  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger\n  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special \"start of sentence\" (SOS)\n  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare\n  images for the model.\n- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly\n  performant image features useful for downstream tasks, such as image classification."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 14e118e1-0e8a-4f93-bc77-3b9ff5621fa9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"14e118e1-0e8a-4f93-bc77-3b9ff5621fa9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"72da8e79-38c2-403e-b0ee-e773bc3fd119\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2dfc2567-ab93-42a7-a142-7ec6c135c881\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"aae7680741c4708254757654b747601a97226a4022ea008b338cc09ad6e89a87\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fe1aa9e0-4c2d-45e6-88fc-2097ed364321\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5fccfa84855df62b8f1903509f9f52776a67d488581354f7e837cf743acf608c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1717, \"end_char_idx\": 3883, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "ref_doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "chroma:document": "The original code can be found\n[here](https://github.com/openai/image-gpt).\n\n## Usage tips\n\n- ImageGPT is almost exactly the same as [GPT-2](gpt2), with the exception that a different activation\n  function is used (namely \"quick gelu\"), and the layer normalization layers don't mean center the inputs. ImageGPT\n  also doesn't have tied input- and output embeddings.\n- As the time- and memory requirements of the attention mechanism of Transformers scales quadratically in the sequence\n  length, the authors pre-trained ImageGPT on smaller input resolutions, such as 32x32 and 64x64. However, feeding a\n  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors\n  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long\n  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger\n  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special \"start of sentence\" (SOS)\n  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare\n  images for the model.\n- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly\n  performant image features useful for downstream tasks, such as image classification. The authors showed that the\n  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as\n  a sklearn logistic regression model for example). This is also referred to as \"linear probing\". Features can be\n  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and\n  then average-pool the hidden states at whatever layer you like.\n- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can\n  use [`ImageGPTForImageClassification`].\n- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also\n  train an XL variant, which they didn't release."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: fe1aa9e0-4c2d-45e6-88fc-2097ed364321
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"fe1aa9e0-4c2d-45e6-88fc-2097ed364321\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"72da8e79-38c2-403e-b0ee-e773bc3fd119\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"14e118e1-0e8a-4f93-bc77-3b9ff5621fa9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"87ae26fec03f36364a0103118620ba77e642585fa3c05ec71d4dab4087cbc87d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2e0a28d-e6a9-407c-a30a-6b1341d93040\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"42c5f4faa6c1ce043a0454ff6de7536d5d55d600c806f46e56d7872e218dedbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2300, \"end_char_idx\": 4630, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "ref_doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "chroma:document": "However, feeding a\n  sequence of 32x32x3=3072 tokens from 0..255 into a Transformer is still prohibitively large. Therefore, the authors\n  applied k-means clustering to the (R,G,B) pixel values with k=512. This way, we only have a 32*32 = 1024-long\n  sequence, but now of integers in the range 0..511. So we are shrinking the sequence length at the cost of a bigger\n  embedding matrix. In other words, the vocabulary size of ImageGPT is 512, + 1 for a special \"start of sentence\" (SOS)\n  token, used at the beginning of every sequence. One can use [`ImageGPTImageProcessor`] to prepare\n  images for the model.\n- Despite being pre-trained entirely unsupervised (i.e. without the use of any labels), ImageGPT produces fairly\n  performant image features useful for downstream tasks, such as image classification. The authors showed that the\n  features in the middle of the network are the most performant, and can be used as-is to train a linear model (such as\n  a sklearn logistic regression model for example). This is also referred to as \"linear probing\". Features can be\n  easily obtained by first forwarding the image through the model, then specifying `output_hidden_states=True`, and\n  then average-pool the hidden states at whatever layer you like.\n- Alternatively, one can further fine-tune the entire model on a downstream dataset, similar to BERT. For this, you can\n  use [`ImageGPTForImageClassification`].\n- ImageGPT comes in different sizes: there's ImageGPT-small, ImageGPT-medium and ImageGPT-large. The authors did also\n  train an XL variant, which they didn't release. The differences in size are summarized in the following table:\n\n| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n|---|---|---|---|---|---|\n| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |\n| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |\n| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |\n| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |\n| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |\n| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ImageGPT."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a2e0a28d-e6a9-407c-a30a-6b1341d93040
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/imagegpt", "title": "ImageGPT", "tokens": 1756, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a2e0a28d-e6a9-407c-a30a-6b1341d93040\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"72da8e79-38c2-403e-b0ee-e773bc3fd119\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7d0f7aaa815ca5acd8f73ac750b1a3c1d3966357b839c371a0c22e3abeb492ca\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fe1aa9e0-4c2d-45e6-88fc-2097ed364321\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/imagegpt\", \"title\": \"ImageGPT\", \"tokens\": 1756, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36d1f8384f88c5f14538bfcda8cd4c7da2b2cf340657789a70d889f9cac5e465\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3813, \"end_char_idx\": 5858, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "ref_doc_id": "72da8e79-38c2-403e-b0ee-e773bc3fd119", "chroma:document": "The authors did also\n  train an XL variant, which they didn't release. The differences in size are summarized in the following table:\n\n| **Model variant** | **Depths** | **Hidden sizes** | **Decoder hidden size** | **Params (M)** | **ImageNet-1k Top 1** |\n|---|---|---|---|---|---|\n| MiT-b0 | [2, 2, 2, 2] | [32, 64, 160, 256] | 256 | 3.7 | 70.5 |\n| MiT-b1 | [2, 2, 2, 2] | [64, 128, 320, 512] | 256 | 14.0 | 78.7 |\n| MiT-b2 | [3, 4, 6, 3] | [64, 128, 320, 512] | 768 | 25.4 | 81.6 |\n| MiT-b3 | [3, 4, 18, 3] | [64, 128, 320, 512] | 768 | 45.2 | 83.1 |\n| MiT-b4 | [3, 8, 27, 3] | [64, 128, 320, 512] | 768 | 62.6 | 83.6 |\n| MiT-b5 | [3, 6, 40, 3] | [64, 128, 320, 512] | 768 | 82.0 | 83.8 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with ImageGPT.\n\n<PipelineTag pipeline=\"image-classification\"/>\n\n- Demo notebooks for ImageGPT can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ImageGPT).\n- [`ImageGPTForImageClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n- See also: [Image classification task guide](../tasks/image_classification)\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ImageGPTConfig\n\n[[autodoc]] ImageGPTConfig\n\n## ImageGPTFeatureExtractor\n\n[[autodoc]] ImageGPTFeatureExtractor\n    - __call__\n\n## ImageGPTImageProcessor\n\n[[autodoc]] ImageGPTImageProcessor\n    - preprocess\n\n## ImageGPTModel\n\n[[autodoc]] ImageGPTModel\n    - forward\n\n## ImageGPTForCausalImageModeling\n\n[[autodoc]] ImageGPTForCausalImageModeling\n    - forward\n\n## ImageGPTForImageClassification\n\n[[autodoc]] ImageGPTForImageClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 14f14103-ffba-433b-bf51-432734ff31d6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nezha", "title": "Nezha", "tokens": 857, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"14f14103-ffba-433b-bf51-432734ff31d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nezha\", \"title\": \"Nezha\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0ab84468-2cfb-43ea-b5fd-976d1af40273\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nezha\", \"title\": \"Nezha\", \"tokens\": 857, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6a0df10602610c8043bee5f55759a8a85a3880618b232e72aff1a48e57aecf4f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2821, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0ab84468-2cfb-43ea-b5fd-976d1af40273", "doc_id": "0ab84468-2cfb-43ea-b5fd-976d1af40273", "ref_doc_id": "0ab84468-2cfb-43ea-b5fd-976d1af40273", "chroma:document": "# Nezha\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe Nezha model was proposed in [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://arxiv.org/abs/1909.00204) by Junqiu Wei et al.\n\nThe abstract from the paper is the following:\n\n*The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks\ndue to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora.\nIn this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed\nrepresentation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks.\nThe current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional\nRelative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy,\nMixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA\nachieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including\nnamed entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti)\nand natural language inference (XNLI).*\n\nThis model was contributed by [sijunhe](https://huggingface.co/sijunhe). The original code can be found [here](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA-PyTorch).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## NezhaConfig\n\n[[autodoc]] NezhaConfig\n\n## NezhaModel\n\n[[autodoc]] NezhaModel\n    - forward\n\n## NezhaForPreTraining\n\n[[autodoc]] NezhaForPreTraining\n    - forward\n\n## NezhaForMaskedLM\n\n[[autodoc]] NezhaForMaskedLM\n    - forward\n\n## NezhaForNextSentencePrediction\n\n[[autodoc]] NezhaForNextSentencePrediction\n    - forward\n\n## NezhaForSequenceClassification\n\n[[autodoc]] NezhaForSequenceClassification\n    - forward\n\n## NezhaForMultipleChoice\n\n[[autodoc]] NezhaForMultipleChoice\n    - forward\n\n## NezhaForTokenClassification\n\n[[autodoc]] NezhaForTokenClassification\n    - forward\n\n## NezhaForQuestionAnswering\n\n[[autodoc]] NezhaForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: b7aed520-5216-47b6-a47a-0b6a5a6b8c93
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"b7aed520-5216-47b6-a47a-0b6a5a6b8c93\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"30162ed6-974d-48ce-a226-5bf2191c49b5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4537d0bafecaafc19090fdc6ad100dad53e1e7b9abcfbbd5e6eeb3cdefb480b5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2e7c925c-4956-44e5-b923-3046f30a233e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4e4b6db29d7225bf740abda8c406d28516077dde573d58a7868566c3b43b491\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "doc_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "ref_doc_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "chroma:document": "# Audio Spectrogram Transformer\n\n## Overview\n\nThe Audio Spectrogram Transformer model was proposed in [AST: Audio Spectrogram Transformer](https://arxiv.org/abs/2104.01778) by Yuan Gong, Yu-An Chung, James Glass.\nThe Audio Spectrogram Transformer applies a [Vision Transformer](vit) to audio, by turning audio into an image (spectrogram). The model obtains state-of-the-art results\nfor audio classification.\n\nThe abstract from the paper is the following:\n\n*In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Audio Spectrogram Transformer architecture. Taken from the <a href=\"https://arxiv.org/abs/2104.01778\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/YuanGongND/ast).\n\n## Usage tips\n\n- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make\nsure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet\nmean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how\nthe authors compute the stats for a downstream dataset.\n- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the\n[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2e7c925c-4956-44e5-b923-3046f30a233e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2e7c925c-4956-44e5-b923-3046f30a233e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"30162ed6-974d-48ce-a226-5bf2191c49b5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4537d0bafecaafc19090fdc6ad100dad53e1e7b9abcfbbd5e6eeb3cdefb480b5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7aed520-5216-47b6-a47a-0b6a5a6b8c93\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7e0ec700be214cd923d500fe7e3eece9a42069d784f7d0d590fe7e4c24967883\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8eaac278-90fa-4cd4-a539-88604c998088\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d901af1d1cdc7253b73f04dc6ef84ddb1b30a0d78d7c3f3bc7d5c81caeba8116\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1651, \"end_char_idx\": 4891, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "doc_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "ref_doc_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "chroma:document": "Taken from the <a href=\"https://arxiv.org/abs/2104.01778\">original paper</a>.</small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr).\nThe original code can be found [here](https://github.com/YuanGongND/ast).\n\n## Usage tips\n\n- When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it's recommended to take care of the input normalization (to make\nsure the input has mean of 0 and std of 0.5). [`ASTFeatureExtractor`] takes care of this. Note that it uses the AudioSet\nmean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how\nthe authors compute the stats for a downstream dataset.\n- Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the\n[PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.\n\n### Using Scaled Dot Product Attention (SDPA)\n\nPyTorch includes a native scaled dot-product attention (SDPA) operator as part of `torch.nn.functional`. This function \nencompasses several implementations that can be applied depending on the inputs and the hardware in use. See the \n[official documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html) \nor the [GPU Inference](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention)\npage for more information.\n\nSDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ASTForAudioClassification\nmodel = ASTForAudioClassification.from_pretrained(\"MIT/ast-finetuned-audioset-10-10-0.4593\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        27 |                                         6 |                      4.5 |\n|            2 |                                        12 |                                         6 |                      2   |\n|            4 |                                        21 |                                         8 |                      2.62 |\n|            8 |                                        40 |                                        14 |                      2.86 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with the Audio Spectrogram Transformer."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 8eaac278-90fa-4cd4-a539-88604c998088
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer", "title": "Audio Spectrogram Transformer", "tokens": 1483, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"8eaac278-90fa-4cd4-a539-88604c998088\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"30162ed6-974d-48ce-a226-5bf2191c49b5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"4537d0bafecaafc19090fdc6ad100dad53e1e7b9abcfbbd5e6eeb3cdefb480b5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e7c925c-4956-44e5-b923-3046f30a233e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer\", \"title\": \"Audio Spectrogram Transformer\", \"tokens\": 1483, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"50d34506a627b52b529a000c792f4a0b21e1353e75c1c78f530c5009b89a1e3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3233, \"end_char_idx\": 5926, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "doc_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "ref_doc_id": "30162ed6-974d-48ce-a226-5bf2191c49b5", "chroma:document": "SDPA is used by default for `torch>=2.1.1` when an implementation is available, but you may also set \n`attn_implementation=\"sdpa\"` in `from_pretrained()` to explicitly request SDPA to be used.\n\n```\nfrom transformers import ASTForAudioClassification\nmodel = ASTForAudioClassification.from_pretrained(\"MIT/ast-finetuned-audioset-10-10-0.4593\", attn_implementation=\"sdpa\", torch_dtype=torch.float16)\n...\n```\n\nFor the best speedups, we recommend loading the model in half-precision (e.g. `torch.float16` or `torch.bfloat16`).\n\nOn a local benchmark (A100-40GB, PyTorch 2.3.0, OS Ubuntu 22.04) with `float32` and `MIT/ast-finetuned-audioset-10-10-0.4593` model, we saw the following speedups during inference.\n\n|   Batch size |   Average inference time (ms), eager mode |   Average inference time (ms), sdpa model |   Speed up, Sdpa / Eager (x) |\n|--------------|-------------------------------------------|-------------------------------------------|------------------------------|\n|            1 |                                        27 |                                         6 |                      4.5 |\n|            2 |                                        12 |                                         6 |                      2   |\n|            4 |                                        21 |                                         8 |                      2.62 |\n|            8 |                                        40 |                                        14 |                      2.86 |\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with the Audio Spectrogram Transformer.\n\n<PipelineTag pipeline=\"audio-classification\"/>\n\n- A notebook illustrating inference with AST for audio classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/AST).\n- [`ASTForAudioClassification`] is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).\n- See also: [Audio classification](../tasks/audio_classification).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## ASTConfig\n\n[[autodoc]] ASTConfig\n\n## ASTFeatureExtractor\n\n[[autodoc]] ASTFeatureExtractor\n    - __call__\n\n## ASTModel\n\n[[autodoc]] ASTModel\n    - forward\n\n## ASTForAudioClassification\n\n[[autodoc]] ASTForAudioClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 739f76f0-b22d-446b-bfe6-42722eabc1e3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mask2former", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"739f76f0-b22d-446b-bfe6-42722eabc1e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f42c0e53-8652-472b-860e-089a73e54f4b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b527f5abeb88edaede15c19e24514c4072fd53c5b2c41a855f6b57db2e64177b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23bb0994-9aa7-481c-b86d-dd4f364f56aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b4286353af35aec9c5e32c51e60427e675483b5392132ad419be9ff6b529f1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3409, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f42c0e53-8652-472b-860e-089a73e54f4b", "doc_id": "f42c0e53-8652-472b-860e-089a73e54f4b", "ref_doc_id": "f42c0e53-8652-472b-860e-089a73e54f4b", "chroma:document": "# Mask2Former\n\n## Overview\n\nThe Mask2Former model was proposed in [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527) by Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar. Mask2Former is a unified framework for panoptic, instance and semantic segmentation and features significant performance and efficiency improvements over [MaskFormer](maskformer).\n\nThe abstract from the paper is the following:\n\n*Image segmentation groups pixels with different semantics, e.g., category or instance membership. Each choice\nof semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mask2former_architecture.jpg\" alt=\"drawing\" width=\"600\"/>\n\n<small> Mask2Former architecture. Taken from the <a href=\"https://arxiv.org/abs/2112.01527\">original paper.</a> </small>\n\nThis model was contributed by [Shivalika Singh](https://huggingface.co/shivi) and [Alara Dirik](https://huggingface.co/adirik). The original code can be found [here](https://github.com/facebookresearch/Mask2Former).\n\n## Usage tips\n\n- Mask2Former uses the same preprocessing and postprocessing steps as [MaskFormer](maskformer). Use [`Mask2FormerImageProcessor`] or [`AutoImageProcessor`] to prepare images and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~Mask2FormerImageProcessor.post_process_semantic_segmentation`] or [`~Mask2FormerImageProcessor.post_process_instance_segmentation`] or [`~Mask2FormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`Mask2FormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mask2Former.\n\n- Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Mask2Former).\n- Scripts for finetuning [`Mask2Former`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 23bb0994-9aa7-481c-b86d-dd4f364f56aa
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mask2former", "title": "Mask2Former", "tokens": 1080, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"23bb0994-9aa7-481c-b86d-dd4f364f56aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f42c0e53-8652-472b-860e-089a73e54f4b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b527f5abeb88edaede15c19e24514c4072fd53c5b2c41a855f6b57db2e64177b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"739f76f0-b22d-446b-bfe6-42722eabc1e3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mask2former\", \"title\": \"Mask2Former\", \"tokens\": 1080, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"ca75991726222526845788e2173e703047901e8f2527d8e439d00638868a5bcc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1832, \"end_char_idx\": 4062, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f42c0e53-8652-472b-860e-089a73e54f4b", "doc_id": "f42c0e53-8652-472b-860e-089a73e54f4b", "ref_doc_id": "f42c0e53-8652-472b-860e-089a73e54f4b", "chroma:document": "The original code can be found [here](https://github.com/facebookresearch/Mask2Former).\n\n## Usage tips\n\n- Mask2Former uses the same preprocessing and postprocessing steps as [MaskFormer](maskformer). Use [`Mask2FormerImageProcessor`] or [`AutoImageProcessor`] to prepare images and optional targets for the model.\n- To get the final segmentation, depending on the task, you can call [`~Mask2FormerImageProcessor.post_process_semantic_segmentation`] or [`~Mask2FormerImageProcessor.post_process_instance_segmentation`] or [`~Mask2FormerImageProcessor.post_process_panoptic_segmentation`]. All three tasks can be solved using [`Mask2FormerForUniversalSegmentation`] output, panoptic segmentation accepts an optional `label_ids_to_fuse` argument to fuse instances of the target object/s (e.g. sky) together.\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with Mask2Former.\n\n- Demo notebooks regarding inference + fine-tuning Mask2Former on custom data can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/Mask2Former).\n- Scripts for finetuning [`Mask2Former`] with [`Trainer`] or [Accelerate](https://huggingface.co/docs/accelerate/index) can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/instance-segmentation).\n\nIf you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it.\nThe resource should ideally demonstrate something new instead of duplicating an existing resource.\n\n## Mask2FormerConfig\n\n[[autodoc]] Mask2FormerConfig\n\n## MaskFormer specific outputs\n\n[[autodoc]] models.mask2former.modeling_mask2former.Mask2FormerModelOutput\n\n[[autodoc]] models.mask2former.modeling_mask2former.Mask2FormerForUniversalSegmentationOutput\n\n## Mask2FormerModel\n\n[[autodoc]] Mask2FormerModel\n    - forward\n\n## Mask2FormerForUniversalSegmentation\n\n[[autodoc]] Mask2FormerForUniversalSegmentation\n    - forward\n\n## Mask2FormerImageProcessor\n\n[[autodoc]] Mask2FormerImageProcessor\n    - preprocess\n    - encode_inputs\n    - post_process_semantic_segmentation\n    - post_process_instance_segmentation\n    - post_process_panoptic_segmentation"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ac582bed-374b-4d26-ac47-097d19f179a3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ac582bed-374b-4d26-ac47-097d19f179a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4d9fab9d-7398-4148-baa1-3f803ec6e8e4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d3f70388fd78df42f23b7f18aacbf5bfd5f6163b88635421df6807ac0e74c275\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ead05675-a617-4ca8-9fe1-3c80d701e778\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d8fa86f6e1ff552031aaa60b886f2bb06b82d080b1961a69ca32028a0f30db33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3117, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4d9fab9d-7398-4148-baa1-3f803ec6e8e4", "doc_id": "4d9fab9d-7398-4148-baa1-3f803ec6e8e4", "ref_doc_id": "4d9fab9d-7398-4148-baa1-3f803ec6e8e4", "chroma:document": "# PatchTSMixer\n\n## Overview\n\nThe PatchTSMixer model was proposed in [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://arxiv.org/pdf/2306.09364.pdf) by Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong and Jayant Kalagnanam.\n\n\nPatchTSMixer is a lightweight time-series modeling approach based on the MLP-Mixer architecture. In this HuggingFace implementation, we provide PatchTSMixer's capabilities to effortlessly facilitate lightweight mixing across patches, channels, and hidden features for effective multivariate time-series modeling. It also supports various attention mechanisms starting from simple gated attention to more complex self-attention blocks that can be customized accordingly. The model can be pretrained and subsequently used for various downstream tasks such as forecasting, classification and regression.\n\n\nThe abstract from the paper is the following:\n\n*TSMixer is a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules designed for multivariate forecasting and representation learning on patched time series. Our model draws inspiration from the success of MLP-Mixer models in computer vision. We demonstrate the challenges involved in adapting Vision MLP-Mixer for time series and introduce empirically validated components to enhance accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a Hybrid channel modeling approach to effectively handle noisy channel interactions and generalization across diverse datasets, a common challenge in existing patch channel-mixing methods. Additionally, a simple gated attention mechanism is introduced in the backbone to prioritize important features. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X).*\n\nThis model was contributed by [ajati](https://huggingface.co/ajati), [vijaye12](https://huggingface.co/vijaye12), \n[gsinthong](https://huggingface.co/gsinthong), [namctin](https://huggingface.co/namctin),\n[wmgifford](https://huggingface.co/wmgifford), [kashif](https://huggingface.co/kashif).\n\n## Usage example\n\nThe code snippet below shows how to randomly initialize a PatchTSMixer model. The model is compatible with the [Trainer API](../trainer.md).\n\n```python\n\nfrom transformers import PatchTSMixerConfig, PatchTSMixerForPrediction\nfrom transformers import Trainer, TrainingArguments,"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: ead05675-a617-4ca8-9fe1-3c80d701e778
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/patchtsmixer", "title": "PatchTSMixer", "tokens": 1090, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"ead05675-a617-4ca8-9fe1-3c80d701e778\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4d9fab9d-7398-4148-baa1-3f803ec6e8e4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d3f70388fd78df42f23b7f18aacbf5bfd5f6163b88635421df6807ac0e74c275\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ac582bed-374b-4d26-ac47-097d19f179a3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/patchtsmixer\", \"title\": \"PatchTSMixer\", \"tokens\": 1090, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"17b190ebb78ff151d3127d3ac9921235c0b6527f96988f1e96f40c717dd96b95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3120, \"end_char_idx\": 4296, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4d9fab9d-7398-4148-baa1-3f803ec6e8e4", "doc_id": "4d9fab9d-7398-4148-baa1-3f803ec6e8e4", "ref_doc_id": "4d9fab9d-7398-4148-baa1-3f803ec6e8e4", "chroma:document": "config = PatchTSMixerConfig(context_length = 512, prediction_length = 96)\nmodel = PatchTSMixerForPrediction(config)\ntrainer = Trainer(model=model, args=training_args, \n            train_dataset=train_dataset,\n            eval_dataset=valid_dataset)\ntrainer.train()\nresults = trainer.evaluate(test_dataset)\n```\n\n## Usage tips\n\nThe model can also be used for time series classification and time series regression. See the respective [`PatchTSMixerForTimeSeriesClassification`] and [`PatchTSMixerForRegression`] classes.\n\n## Resources\n\n- A blog post explaining PatchTSMixer in depth can be found [here](https://huggingface.co/blog/patchtsmixer). The blog can also be opened in Google Colab.\n\n## PatchTSMixerConfig\n\n[[autodoc]] PatchTSMixerConfig\n\n\n## PatchTSMixerModel\n\n[[autodoc]] PatchTSMixerModel\n    - forward\n\n\n## PatchTSMixerForPrediction\n\n[[autodoc]] PatchTSMixerForPrediction\n    - forward\n\n\n## PatchTSMixerForTimeSeriesClassification\n\n[[autodoc]] PatchTSMixerForTimeSeriesClassification\n    - forward\n\n\n## PatchTSMixerForPretraining\n\n[[autodoc]] PatchTSMixerForPretraining\n    - forward\n\n\n## PatchTSMixerForRegression\n\n[[autodoc]] PatchTSMixerForRegression\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 42d7143c-7387-46b6-82aa-6ee611e3511a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"42d7143c-7387-46b6-82aa-6ee611e3511a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7de0f52a-0824-4862-b644-bf87bca679ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36d0c11a54941a17706a451e10497bd1721a06d28b03ba5569b5833001f3a7d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3344de01-35f2-4eff-bf4a-4886c3c87914\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1dd7787d4c5d5d1358e50d40eeb33a44e315385e0003840c427830e461f7a0f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3117, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "ref_doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "chroma:document": "# GPTBigCode\n\n## Overview\n\nThe GPTBigCode model was proposed in [SantaCoder: don't reach for the stars!](https://arxiv.org/abs/2301.03988) by BigCode. The listed authors are: Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo Garc\u00eda del R\u00edo, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra.\n\nThe abstract from the paper is the following:\n\n*The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the collaboration until December 2022, outlining the current state of the Personally Identifiable Information (PII) redaction pipeline, the experiments conducted to de-risk the model architecture, and the experiments investigating better preprocessing methods for the training data. We train 1.1B parameter models on the Java, JavaScript, and Python subsets of The Stack and evaluate them on the MultiPL-E text-to-code benchmark. We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at [this https URL.](https://huggingface.co/bigcode)*\n\nThe model is an optimized [GPT2 model](https://huggingface.co/docs/transformers/model_doc/gpt2) with support for Multi-Query Attention.\n\n## Implementation details\n\nThe main differences compared to GPT2.\n- Added support for Multi-Query Attention.\n- Use `gelu_pytorch_tanh` instead of classic `gelu`.\n- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).\n- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).\n- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`\n- Cache the attention mask value to avoid recreating it every time.\n- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.\n- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.\n- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3344de01-35f2-4eff-bf4a-4886c3c87914
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3344de01-35f2-4eff-bf4a-4886c3c87914\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7de0f52a-0824-4862-b644-bf87bca679ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36d0c11a54941a17706a451e10497bd1721a06d28b03ba5569b5833001f3a7d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"42d7143c-7387-46b6-82aa-6ee611e3511a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"93d1a17298e55365a04d4550ab2ea0b4fe2097b8a9c758dadc2c6de8b0b880d1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9767c076-9f9d-435c-9d59-6a579d9cae67\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0d3cbefe0687d8c467dd8b5332bbc78467db7c65dda39530751c69ff35c60ab5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1483, \"end_char_idx\": 3965, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "ref_doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "chroma:document": "We find that more aggressive filtering of near-duplicates can further boost performance and, surprisingly, that selecting files from repositories with 5+ GitHub stars deteriorates performance significantly. Our best model outperforms previous open-source multilingual code generation models (InCoder-6.7B and CodeGen-Multi-2.7B) in both left-to-right generation and infilling on the Java, JavaScript, and Python portions of MultiPL-E, despite being a substantially smaller model. All models are released under an OpenRAIL license at [this https URL.](https://huggingface.co/bigcode)*\n\nThe model is an optimized [GPT2 model](https://huggingface.co/docs/transformers/model_doc/gpt2) with support for Multi-Query Attention.\n\n## Implementation details\n\nThe main differences compared to GPT2.\n- Added support for Multi-Query Attention.\n- Use `gelu_pytorch_tanh` instead of classic `gelu`.\n- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).\n- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).\n- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`\n- Cache the attention mask value to avoid recreating it every time.\n- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.\n- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.\n- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)\n- Use the memory layout (self.num_heads, 3, self.head_dim) instead of `(3, self.num_heads, self.head_dim)` for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible with the original openai-community/gpt2 model).\n\nYou can read more about the optimizations in the [original pull request](https://github.com/huggingface/transformers/pull/22575)\n\n## Combining Starcoder and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9767c076-9f9d-435c-9d59-6a579d9cae67
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9767c076-9f9d-435c-9d59-6a579d9cae67\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7de0f52a-0824-4862-b644-bf87bca679ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36d0c11a54941a17706a451e10497bd1721a06d28b03ba5569b5833001f3a7d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3344de01-35f2-4eff-bf4a-4886c3c87914\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"6acb3998b1e591c46248f483e222f1fbbade6e32f2e972654114b42b8abcd07a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6606016f-3b45-4b85-9729-df55ab4e5927\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc0bfe69c72e629cb05830c35b3b9cb24f0283d0bdaaa27237d12bbde2a8b099\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2271, \"end_char_idx\": 5241, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "ref_doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "chroma:document": "- Added support for Multi-Query Attention.\n- Use `gelu_pytorch_tanh` instead of classic `gelu`.\n- Avoid unnecessary synchronizations (this has since been added to GPT2 in #20061, but wasn't in the reference codebase).\n- Use Linear layers instead of Conv1D (good speedup but makes the checkpoints incompatible).\n- Merge `_attn` and `_upcast_and_reordered_attn`. Always merge the matmul with scaling. Rename `reorder_and_upcast_attn`->`attention_softmax_in_fp32`\n- Cache the attention mask value to avoid recreating it every time.\n- Use jit to fuse the attention fp32 casting, masking, softmax, and scaling.\n- Combine the attention and causal masks into a single one, pre-computed for the whole model instead of every layer.\n- Merge the key and value caches into one (this changes the format of layer_past/ present, does it risk creating problems?)\n- Use the memory layout (self.num_heads, 3, self.head_dim) instead of `(3, self.num_heads, self.head_dim)` for the QKV tensor with MHA. (prevents an overhead with the merged key and values, but makes the checkpoints incompatible with the original openai-community/gpt2 model).\n\nYou can read more about the optimizations in the [original pull request](https://github.com/huggingface/transformers/pull/22575)\n\n## Combining Starcoder and Flash Attention 2\n\nFirst, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"bigcode/gpt_bigcode-santacoder\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"bigcode/gpt_bigcode-santacoder\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'def hello_world():\\n    print(\"hello world\")\\n\\nif __name__ == \"__main__\":\\n    print(\"hello world\")\\n<|endoftext|>'\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `bigcode/starcoder` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/starcoder-speedup.png\">\n</div>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6606016f-3b45-4b85-9729-df55ab4e5927
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gpt_bigcode", "title": "GPTBigCode", "tokens": 1538, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6606016f-3b45-4b85-9729-df55ab4e5927\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7de0f52a-0824-4862-b644-bf87bca679ee\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"36d0c11a54941a17706a451e10497bd1721a06d28b03ba5569b5833001f3a7d9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9767c076-9f9d-435c-9d59-6a579d9cae67\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gpt_bigcode\", \"title\": \"GPTBigCode\", \"tokens\": 1538, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"202b81c88788d644668a00fc8083d49ce005aaad29a404684cc7b6cb65106c6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3690, \"end_char_idx\": 5631, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "ref_doc_id": "7de0f52a-0824-4862-b644-bf87bca679ee", "chroma:document": "```bash\npip install -U flash-attn --no-build-isolation\n```\n\nMake also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. `torch.float16``)\n\nTo load and run a model using Flash Attention 2, refer to the snippet below:\n\n```python\n>>> import torch\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n>>> device = \"cuda\" # the device to load the model onto\n\n>>> model = AutoModelForCausalLM.from_pretrained(\"bigcode/gpt_bigcode-santacoder\", torch_dtype=torch.float16, attn_implementation=\"flash_attention_2\")\n>>> tokenizer = AutoTokenizer.from_pretrained(\"bigcode/gpt_bigcode-santacoder\")\n\n>>> prompt = \"def hello_world():\"\n\n>>> model_inputs = tokenizer([prompt], return_tensors=\"pt\").to(device)\n>>> model.to(device)\n\n>>> generated_ids = model.generate(**model_inputs, max_new_tokens=30, do_sample=False)\n>>> tokenizer.batch_decode(generated_ids)[0]\n'def hello_world():\\n    print(\"hello world\")\\n\\nif __name__ == \"__main__\":\\n    print(\"hello world\")\\n<|endoftext|>'\n```\n\n### Expected speedups\n\nBelow is a expected speedup diagram that compares pure inference time between the native implementation in transformers using `bigcode/starcoder` checkpoint and the Flash Attention 2 version of the model using two different sequence lengths.\n\n<div style=\"text-align: center\">\n<img src=\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/starcoder-speedup.png\">\n</div>\n\n\n## GPTBigCodeConfig\n\n[[autodoc]] GPTBigCodeConfig\n\n## GPTBigCodeModel\n\n[[autodoc]] GPTBigCodeModel\n    - forward\n\n## GPTBigCodeForCausalLM\n\n[[autodoc]] GPTBigCodeForCausalLM\n    - forward\n\n## GPTBigCodeForSequenceClassification\n\n[[autodoc]] GPTBigCodeForSequenceClassification\n    - forward\n\n## GPTBigCodeForTokenClassification\n\n[[autodoc]] GPTBigCodeForTokenClassification\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 6901e7a6-ba1a-4214-b243-f23d617a7028
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/nystromformer", "title": "Nystr\u00f6mformer", "tokens": 796, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"6901e7a6-ba1a-4214-b243-f23d617a7028\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nystromformer\", \"title\": \"Nystr\\u00f6mformer\", \"tokens\": 796, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"958b8ad7-b265-4829-9dd0-8d7e82fcff21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/nystromformer\", \"title\": \"Nystr\\u00f6mformer\", \"tokens\": 796, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"75138ee827207eae5b096c44b1dfbf2b1a43e70772614c3deeb27bcdf44b607e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2705, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "958b8ad7-b265-4829-9dd0-8d7e82fcff21", "doc_id": "958b8ad7-b265-4829-9dd0-8d7e82fcff21", "ref_doc_id": "958b8ad7-b265-4829-9dd0-8d7e82fcff21", "chroma:document": "# Nystr\u00f6mformer\n\n## Overview\n\nThe Nystr\u00f6mformer model was proposed in [*Nystr\u00f6mformer: A Nystr\u00f6m-Based Algorithm for Approximating Self-Attention*](https://arxiv.org/abs/2102.03902) by Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn\nFung, Yin Li, and Vikas Singh.\n\nThe abstract from the paper is the following:\n\n*Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component\nthat drives the impressive performance of Transformers is the self-attention mechanism that encodes the influence or\ndependence of other tokens on each specific token. While beneficial, the quadratic complexity of self-attention on the\ninput sequence length has limited its application to longer sequences -- a topic being actively studied in the\ncommunity. To address this limitation, we propose Nystr\u00f6mformer -- a model that exhibits favorable scalability as a\nfunction of sequence length. Our idea is based on adapting the Nystr\u00f6m method to approximate standard self-attention\nwith O(n) complexity. The scalability of Nystr\u00f6mformer enables application to longer sequences with thousands of\ntokens. We perform evaluations on multiple downstream tasks on the GLUE benchmark and IMDB reviews with standard\nsequence length, and find that our Nystr\u00f6mformer performs comparably, or in a few cases, even slightly better, than\nstandard self-attention. On longer sequence tasks in the Long Range Arena (LRA) benchmark, Nystr\u00f6mformer performs\nfavorably relative to other efficient self-attention methods. Our code is available at this https URL.*\n\nThis model was contributed by [novice03](https://huggingface.co/novice03). The original code can be found [here](https://github.com/mlpen/Nystromformer).\n\n## Resources\n\n- [Text classification task guide](../tasks/sequence_classification)\n- [Token classification task guide](../tasks/token_classification)\n- [Question answering task guide](../tasks/question_answering)\n- [Masked language modeling task guide](../tasks/masked_language_modeling)\n- [Multiple choice task guide](../tasks/multiple_choice)\n\n## NystromformerConfig\n\n[[autodoc]] NystromformerConfig\n\n## NystromformerModel\n\n[[autodoc]] NystromformerModel\n    - forward\n\n## NystromformerForMaskedLM\n\n[[autodoc]] NystromformerForMaskedLM\n    - forward\n\n## NystromformerForSequenceClassification\n\n[[autodoc]] NystromformerForSequenceClassification\n    - forward\n\n## NystromformerForMultipleChoice\n\n[[autodoc]] NystromformerForMultipleChoice\n    - forward\n\n## NystromformerForTokenClassification\n\n[[autodoc]] NystromformerForTokenClassification\n    - forward\n\n## NystromformerForQuestionAnswering\n\n[[autodoc]] NystromformerForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 4aa42f48-faea-42a6-951e-f2de2f04b416
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/glpn", "title": "GLPN", "tokens": 895, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"4aa42f48-faea-42a6-951e-f2de2f04b416\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/glpn\", \"title\": \"GLPN\", \"tokens\": 895, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c7470b8d-4132-4cf2-94dc-d9ac88fad9a1\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/glpn\", \"title\": \"GLPN\", \"tokens\": 895, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"d479199ee4b62f4b1e6fcac4b094065de7df76c6bfcb99b0301153651a964622\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c7470b8d-4132-4cf2-94dc-d9ac88fad9a1", "doc_id": "c7470b8d-4132-4cf2-94dc-d9ac88fad9a1", "ref_doc_id": "c7470b8d-4132-4cf2-94dc-d9ac88fad9a1", "chroma:document": "# GLPN\n\n<Tip>\n\nThis is a recently introduced model so the API hasn't been tested extensively. There may be some bugs or slight\nbreaking changes to fix it in the future. If you see something strange, file a [Github Issue](https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title).\n\n</Tip>\n\n## Overview\n\nThe GLPN model was proposed in [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://arxiv.org/abs/2201.07436)  by Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim.\nGLPN combines [SegFormer](segformer)'s hierarchical mix-Transformer with a lightweight decoder for monocular depth estimation. The proposed decoder shows better performance than the previously proposed decoders, with considerably\nless computational complexity.\n\nThe abstract from the paper is the following:\n\n*Depth estimation from a single image is an important task that can be applied to various fields in computer vision, and has grown rapidly with the development of convolutional neural networks. In this paper, we propose a novel structure and training strategy for monocular depth estimation to further improve the prediction accuracy of the network. We deploy a hierarchical transformer encoder to capture and convey the global context, and design a lightweight yet powerful decoder to generate an estimated depth map while considering local connectivity. By constructing connected paths between multi-scale local features and the global decoding stream with our proposed selective feature fusion module, the network can integrate both representations and recover fine details. In addition, the proposed decoder shows better performance than the previously proposed decoders, with considerably less computational complexity. Furthermore, we improve the depth-specific augmentation method by utilizing an important observation in depth estimation to enhance the model. Our network achieves state-of-the-art performance over the challenging depth dataset NYU Depth V2. Extensive experiments have been conducted to validate and show the effectiveness of the proposed approach. Finally, our model shows better generalisation ability and robustness than other comparative models.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> Summary of the approach. Taken from the <a href=\"https://arxiv.org/abs/2201.07436\" target=\"_blank\">original paper</a>. </small>\n\nThis model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/vinvino02/GLPDepth).\n\n## Resources\n\nA list of official Hugging Face and community (indicated by \ud83c\udf0e) resources to help you get started with GLPN.\n\n- Demo notebooks for [`GLPNForDepthEstimation`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/GLPN).\n- [Monocular depth estimation task guide](../tasks/monocular_depth_estimation)\n\n## GLPNConfig\n\n[[autodoc]] GLPNConfig\n\n## GLPNFeatureExtractor\n\n[[autodoc]] GLPNFeatureExtractor\n    - __call__\n\n## GLPNImageProcessor\n\n[[autodoc]] GLPNImageProcessor\n    - preprocess\n\n## GLPNModel\n\n[[autodoc]] GLPNModel\n    - forward\n\n## GLPNForDepthEstimation\n\n[[autodoc]] GLPNForDepthEstimation\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 73486d81-3ceb-418b-a0c8-9a3237d6bdfb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/gemma", "title": "Gemma", "tokens": 661, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"73486d81-3ceb-418b-a0c8-9a3237d6bdfb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma\", \"title\": \"Gemma\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"495878fd-a3cd-4702-804b-ca9b13b0ade3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/gemma\", \"title\": \"Gemma\", \"tokens\": 661, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"b9c670d892c7c4b40df5d78aa28ab026ae671760dbe579d273a30b3601586be7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1970, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "495878fd-a3cd-4702-804b-ca9b13b0ade3", "doc_id": "495878fd-a3cd-4702-804b-ca9b13b0ade3", "ref_doc_id": "495878fd-a3cd-4702-804b-ca9b13b0ade3", "chroma:document": "# Gemma\n\n## Overview\n\nThe Gemma model was proposed in [Gemma: Open Models Based on Gemini Technology and Research](https://blog.google/technology/developers/gemma-open-models/) by Gemma Team, Google.\nGemma models are trained on 6T tokens, and released with 2 versions, 2b and 7b.\n\nThe abstract from the paper is the following:\n\n*This work introduces Gemma, a new family of open language models demonstrating strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of our model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations*\n\nTips:\n\n- The original checkpoints can be converted using the conversion script `src/transformers/models/gemma/convert_gemma_weights_to_hf.py` \n\nThis model was contributed by [Arthur Zucker](https://huggingface.co/ArthurZ), [Younes Belkada](https://huggingface.co/ybelkada), [Sanchit Gandhi](https://huggingface.co/sanchit-gandhi), [Pedro Cuenca](https://huggingface.co/pcuenq).\n\n\n## GemmaConfig\n\n[[autodoc]] GemmaConfig\n\n## GemmaTokenizer\n\n[[autodoc]] GemmaTokenizer\n\n\n## GemmaTokenizerFast\n\n[[autodoc]] GemmaTokenizerFast\n\n## GemmaModel\n\n[[autodoc]] GemmaModel\n    - forward\n\n## GemmaForCausalLM\n\n[[autodoc]] GemmaForCausalLM\n    - forward\n\n## GemmaForSequenceClassification\n\n[[autodoc]] GemmaForSequenceClassification\n    - forward\n\n## GemmaForTokenClassification\n\n[[autodoc]] GemmaForTokenClassification\n    - forward\n\n## FlaxGemmaModel\n\n[[autodoc]] FlaxGemmaModel\n    - __call__\n\n## FlaxGemmaForCausalLM\n\n[[autodoc]] FlaxGemmaForCausalLM\n    - __call__"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 717c197f-f46a-429b-9984-6895b29d70c2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/wavlm", "title": "WavLM", "tokens": 859, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"717c197f-f46a-429b-9984-6895b29d70c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wavlm\", \"title\": \"WavLM\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0e93a5f3-6d5c-4a4f-a69d-c049d81f54f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/wavlm\", \"title\": \"WavLM\", \"tokens\": 859, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"9ae0e2ea940e385f5991e945c609af53c26fdb73037152ef8cfc445c0d21d427\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2851, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0e93a5f3-6d5c-4a4f-a69d-c049d81f54f6", "doc_id": "0e93a5f3-6d5c-4a4f-a69d-c049d81f54f6", "ref_doc_id": "0e93a5f3-6d5c-4a4f-a69d-c049d81f54f6", "chroma:document": "# WavLM\n\n## Overview\n\nThe WavLM model was proposed in [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://arxiv.org/abs/2110.13900) by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen,\nJinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu,\nMichael Zeng, Furu Wei.\n\nThe abstract from the paper is the following:\n\n*Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been\nattempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker\nidentity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is\nchallenging. In this paper, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks.\nWavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity\npreservation. We first equip the Transformer structure with gated relative position bias to improve its capability on\nrecognition tasks. For better speaker discrimination, we propose an utterance mixing training strategy, where\nadditional overlapped utterances are created unsupervisedly and incorporated during model training. Lastly, we scale up\nthe training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB\nbenchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks.*\n\nRelevant checkpoints can be found under https://huggingface.co/models?other=wavlm.\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/microsoft/unilm/tree/master/wavlm).\n\n## Usage tips\n\n- WavLM is a speech model that accepts a float array corresponding to the raw waveform of the speech signal. Please use\n  [`Wav2Vec2Processor`] for the feature extraction.\n- WavLM model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded\n  using [`Wav2Vec2CTCTokenizer`].\n- WavLM performs especially well on speaker verification, speaker identification, and speaker diarization tasks.\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## WavLMConfig\n\n[[autodoc]] WavLMConfig\n\n## WavLMModel\n\n[[autodoc]] WavLMModel\n    - forward\n\n## WavLMForCTC\n\n[[autodoc]] WavLMForCTC\n    - forward\n\n## WavLMForSequenceClassification\n\n[[autodoc]] WavLMForSequenceClassification\n    - forward\n\n## WavLMForAudioFrameClassification\n\n[[autodoc]] WavLMForAudioFrameClassification\n    - forward\n\n## WavLMForXVector\n\n[[autodoc]] WavLMForXVector\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: a73ee430-90b5-4f04-ba14-c5af4b02469f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/unispeech-sat", "title": "UniSpeech-SAT", "tokens": 891, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"a73ee430-90b5-4f04-ba14-c5af4b02469f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech-sat\", \"title\": \"UniSpeech-SAT\", \"tokens\": 891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f7861cde-4e94-4905-a21d-f944fb7c9aa2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/unispeech-sat\", \"title\": \"UniSpeech-SAT\", \"tokens\": 891, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"02f7755648c36bca122f5d2047de8fe1b9ce85a5acda8a74e1dd08c2f7bbd5fc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3262, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f7861cde-4e94-4905-a21d-f944fb7c9aa2", "doc_id": "f7861cde-4e94-4905-a21d-f944fb7c9aa2", "ref_doc_id": "f7861cde-4e94-4905-a21d-f944fb7c9aa2", "chroma:document": "# UniSpeech-SAT\n\n## Overview\n\nThe UniSpeech-SAT model was proposed in [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware\nPre-Training](https://arxiv.org/abs/2110.05752) by Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen,\nShujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu .\n\nThe abstract from the paper is the following:\n\n*Self-supervised learning (SSL) is a long-standing goal for speech processing, since it utilizes large-scale unlabeled\ndata and avoids extensive human labeling. Recent years witness great successes in applying self-supervised learning in\nspeech recognition, while limited exploration was attempted in applying SSL for modeling speaker characteristics. In\nthis paper, we aim to improve the existing SSL framework for speaker representation learning. Two methods are\nintroduced for enhancing the unsupervised speaker information extraction. First, we apply the multi-task learning to\nthe current SSL framework, where we integrate the utterance-wise contrastive loss with the SSL objective function.\nSecond, for better speaker discrimination, we propose an utterance mixing strategy for data augmentation, where\nadditional overlapped utterances are created unsupervisedly and incorporate during training. We integrate the proposed\nmethods into the HuBERT framework. Experiment results on SUPERB benchmark show that the proposed system achieves\nstate-of-the-art performance in universal representation learning, especially for speaker identification oriented\ntasks. An ablation study is performed verifying the efficacy of each proposed method. Finally, we scale up training\ndataset to 94 thousand hours public audio data and achieve further performance improvement in all SUPERB tasks.*\n\nThis model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The Authors' code can be\nfound [here](https://github.com/microsoft/UniSpeech/tree/main/UniSpeech-SAT).\n\n## Usage tips\n\n- UniSpeechSat is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.\n  Please use [`Wav2Vec2Processor`] for the feature extraction.\n- UniSpeechSat model can be fine-tuned using connectionist temporal classification (CTC) so the model output has to be\n  decoded using [`Wav2Vec2CTCTokenizer`].\n- UniSpeechSat performs especially well on speaker verification, speaker identification, and speaker diarization tasks.\n\n## Resources\n\n- [Audio classification task guide](../tasks/audio_classification)\n- [Automatic speech recognition task guide](../tasks/asr)\n\n## UniSpeechSatConfig\n\n[[autodoc]] UniSpeechSatConfig\n\n## UniSpeechSat specific outputs\n\n[[autodoc]] models.unispeech_sat.modeling_unispeech_sat.UniSpeechSatForPreTrainingOutput\n\n## UniSpeechSatModel\n\n[[autodoc]] UniSpeechSatModel\n    - forward\n\n## UniSpeechSatForCTC\n\n[[autodoc]] UniSpeechSatForCTC\n    - forward\n\n## UniSpeechSatForSequenceClassification\n\n[[autodoc]] UniSpeechSatForSequenceClassification\n    - forward\n\n## UniSpeechSatForAudioFrameClassification\n\n[[autodoc]] UniSpeechSatForAudioFrameClassification\n    - forward\n\n## UniSpeechSatForXVector\n\n[[autodoc]] UniSpeechSatForXVector\n    - forward\n\n## UniSpeechSatForPreTraining\n\n[[autodoc]] UniSpeechSatForPreTraining\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: cc71757a-b407-442e-a30e-253a09ac3b2c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpr", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"cc71757a-b407-442e-a30e-253a09ac3b2c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c5f02a7c-f1f6-4ce2-94c2-67273de01fd5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f8efa39b7e7eb3df5ff1339be56a560618ea528dc8daba9c5a6d5408f7d394f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30540b09-34ad-4755-a0e4-080d524970b9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"21851b0c561b8518650206f11e00571ae7bd1cd5fd205a0df870c75ce5f692e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c5f02a7c-f1f6-4ce2-94c2-67273de01fd5", "doc_id": "c5f02a7c-f1f6-4ce2-94c2-67273de01fd5", "ref_doc_id": "c5f02a7c-f1f6-4ce2-94c2-67273de01fd5", "chroma:document": "# DPR\n\n<div class=\"flex flex-wrap space-x-1\">\n<a href=\"https://huggingface.co/models?filter=dpr\">\n<img alt=\"Models\" src=\"https://img.shields.io/badge/All_model_pages-dpr-blueviolet\">\n</a>\n<a href=\"https://huggingface.co/spaces/docs-demos/dpr-question_encoder-bert-base-multilingual\">\n<img alt=\"Spaces\" src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue\">\n</a>\n</div>\n\n## Overview\n\nDense Passage Retrieval (DPR) is a set of tools and models for state-of-the-art open-domain Q&A research. It was\nintroduced in [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) by\nVladimir Karpukhin, Barlas O\u011fuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.\n\nThe abstract from the paper is the following:\n\n*Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional\nsparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can\nbe practically implemented using dense representations alone, where embeddings are learned from a small number of\nquestions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,\nour dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage\nretrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA\nbenchmarks.*\n\nThis model was contributed by [lhoestq](https://huggingface.co/lhoestq). The original code can be found [here](https://github.com/facebookresearch/DPR).\n\n## Usage tips\n\n- DPR consists in three models:\n\n    * Question encoder: encode questions as vectors\n    * Context encoder: encode contexts as vectors\n    * Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question)."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 30540b09-34ad-4755-a0e4-080d524970b9
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/dpr", "title": "DPR", "tokens": 953, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"30540b09-34ad-4755-a0e4-080d524970b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c5f02a7c-f1f6-4ce2-94c2-67273de01fd5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1f8efa39b7e7eb3df5ff1339be56a560618ea528dc8daba9c5a6d5408f7d394f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cc71757a-b407-442e-a30e-253a09ac3b2c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/dpr\", \"title\": \"DPR\", \"tokens\": 953, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"a7b57eb0f238aaddfd935732394a502320338d8b962779f8e04e9f39e9551e52\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 512, \"end_char_idx\": 3067, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c5f02a7c-f1f6-4ce2-94c2-67273de01fd5", "doc_id": "c5f02a7c-f1f6-4ce2-94c2-67273de01fd5", "ref_doc_id": "c5f02a7c-f1f6-4ce2-94c2-67273de01fd5", "chroma:document": "It was\nintroduced in [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) by\nVladimir Karpukhin, Barlas O\u011fuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.\n\nThe abstract from the paper is the following:\n\n*Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional\nsparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can\nbe practically implemented using dense representations alone, where embeddings are learned from a small number of\nquestions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets,\nour dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage\nretrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA\nbenchmarks.*\n\nThis model was contributed by [lhoestq](https://huggingface.co/lhoestq). The original code can be found [here](https://github.com/facebookresearch/DPR).\n\n## Usage tips\n\n- DPR consists in three models:\n\n    * Question encoder: encode questions as vectors\n    * Context encoder: encode contexts as vectors\n    * Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question).\n\n## DPRConfig\n\n[[autodoc]] DPRConfig\n\n## DPRContextEncoderTokenizer\n\n[[autodoc]] DPRContextEncoderTokenizer\n\n## DPRContextEncoderTokenizerFast\n\n[[autodoc]] DPRContextEncoderTokenizerFast\n\n## DPRQuestionEncoderTokenizer\n\n[[autodoc]] DPRQuestionEncoderTokenizer\n\n## DPRQuestionEncoderTokenizerFast\n\n[[autodoc]] DPRQuestionEncoderTokenizerFast\n\n## DPRReaderTokenizer\n\n[[autodoc]] DPRReaderTokenizer\n\n## DPRReaderTokenizerFast\n\n[[autodoc]] DPRReaderTokenizerFast\n\n## DPR specific outputs\n\n[[autodoc]] models.dpr.modeling_dpr.DPRContextEncoderOutput\n\n[[autodoc]] models.dpr.modeling_dpr.DPRQuestionEncoderOutput\n\n[[autodoc]] models.dpr.modeling_dpr.DPRReaderOutput\n\n<frameworkcontent>\n<pt>\n\n## DPRContextEncoder\n\n[[autodoc]] DPRContextEncoder\n    - forward\n\n## DPRQuestionEncoder\n\n[[autodoc]] DPRQuestionEncoder\n    - forward\n\n## DPRReader\n\n[[autodoc]] DPRReader\n    - forward\n\n</pt>\n<tf>\n\n## TFDPRContextEncoder\n\n[[autodoc]] TFDPRContextEncoder\n    - call\n\n## TFDPRQuestionEncoder\n\n[[autodoc]] TFDPRQuestionEncoder\n    - call\n\n## TFDPRReader\n\n[[autodoc]] TFDPRReader\n    - call\n\n</tf>\n</frameworkcontent>"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d1e4d156-32ad-4307-b254-5fd36e38ec11
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/realm", "title": "REALM", "tokens": 856, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d1e4d156-32ad-4307-b254-5fd36e38ec11\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/realm\", \"title\": \"REALM\", \"tokens\": 856, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3384634c-b9af-4929-a9f6-4c6ea3568949\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/realm\", \"title\": \"REALM\", \"tokens\": 856, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"1028ee5aa34ff18639c5f0e77273fd6edc31233aab2adad2fcc1010ab1f002ac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3384634c-b9af-4929-a9f6-4c6ea3568949", "doc_id": "3384634c-b9af-4929-a9f6-4c6ea3568949", "ref_doc_id": "3384634c-b9af-4929-a9f6-4c6ea3568949", "chroma:document": "# REALM\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe REALM model was proposed in [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909) by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang. It's a\nretrieval-augmented language model that firstly retrieves documents from a textual knowledge corpus and then\nutilizes retrieved documents to process question answering tasks.\n\nThe abstract from the paper is the following:\n\n*Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks\nsuch as question answering. However, this knowledge is stored implicitly in the parameters of a neural network,\nrequiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we\naugment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend\nover documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the\nfirst time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language\nmodeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We\ndemonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the\nchallenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both\nexplicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous\nmethods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as\ninterpretability and modularity.*\n\nThis model was contributed by [qqaatw](https://huggingface.co/qqaatw). The original code can be found\n[here](https://github.com/google-research/language/tree/master/language/realm).\n\n## RealmConfig\n\n[[autodoc]] RealmConfig\n\n## RealmTokenizer\n\n[[autodoc]] RealmTokenizer\n    - build_inputs_with_special_tokens\n    - get_special_tokens_mask\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n    - batch_encode_candidates\n\n## RealmTokenizerFast\n\n[[autodoc]] RealmTokenizerFast\n    - batch_encode_candidates\n\n## RealmRetriever\n\n[[autodoc]] RealmRetriever\n\n## RealmEmbedder\n\n[[autodoc]] RealmEmbedder\n    - forward\n\n## RealmScorer\n\n[[autodoc]] RealmScorer\n    - forward\n\n## RealmKnowledgeAugEncoder\n\n[[autodoc]] RealmKnowledgeAugEncoder\n    - forward\n\n## RealmReader\n\n[[autodoc]] RealmReader\n    - forward\n\n## RealmForOpenQA\n\n[[autodoc]] RealmForOpenQA\n    - block_embedding_to\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 9aa90f93-001f-4594-beba-a94dbbb95f60
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"9aa90f93-001f-4594-beba-a94dbbb95f60\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b8cc3e86-bca2-43dd-9c29-3b8e14f4e083\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a0b992f56c7b17508f2a9e9e451bff8068f4fc27d81ff60ec3863b9fa4e40a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c73fb89-6d56-4f15-a8c6-d8f2588c5eb2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5afae6ed9e4f976602b8ef2c2b78cf83f4a5d765f2c27de0b3f692d07bcc6e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3354, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "ref_doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "chroma:document": "# Video-LLaVA\n\n## Overview\n\nVideo-LLaVa is an open-source multimodal LLM trained by fine-tuning LlamA/Vicuna on multimodal instruction-following data generated by Llava1.5 and VideChat. It is an auto-regressive language model, based on the transformer architecture. Video-LLaVa unifies visual representations to the language feature space, and enables an LLM to perform visual reasoning capabilities on both images and videos simultaneously.\n\n\nThe Video-LLaVA model was proposed in [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://arxiv.org/abs/2311.10122) by Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munang Ning, Peng Jin, Li Yuan.\n\nThe abstract from the paper is the following:\n\n*The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in\nvisual-language understanding. Most existing approaches\nencode images and videos into separate feature spaces,\nwhich are then fed as inputs to large language models.\nHowever, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it\nbecomes challenging for a Large Language Model (LLM)\nto learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA,\nwhich learns from a mixed dataset of images and videos,\nmutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4\nimage benchmark toolkits. Additionally, our Video-LLaVA\nalso outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%,\nand 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that\nVideo-LLaVA mutually benefits images and videos within\na unified visual representation, outperforming models designed specifically for images or videos. We aim for this\nwork to provide modest insights into the multi-modal inputs\nfor the LLM*\n\n## Usage tips:\n\n- We advise users to use padding_side=\"left\" when computing batched generation as it leads to more accurate results. Simply make sure to call processor.tokenizer.padding_side = \"left\" before generating.\n\n- Note the model has not been explicitly trained to process multiple images/videos in the same prompt, although this is technically possible, you may experience inaccurate results.\n\n- Note that the video inputs should have exactly 8 frames at the input, since the models were trained in that setting. \n\nThis model was contributed by [RaushanTurganbay](https://huggingface.co/RaushanTurganbay).\nThe original code can be found [here](https://github.com/PKU-YuanGroup/Video-LLaVA).\n\n\n## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 1c73fb89-6d56-4f15-a8c6-d8f2588c5eb2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"1c73fb89-6d56-4f15-a8c6-d8f2588c5eb2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b8cc3e86-bca2-43dd-9c29-3b8e14f4e083\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a0b992f56c7b17508f2a9e9e451bff8068f4fc27d81ff60ec3863b9fa4e40a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9aa90f93-001f-4594-beba-a94dbbb95f60\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"e1b4b65f3acc55f38a6c3af09b6bf4f48b1f56429368678a6ff948a0c3adc738\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3d4355ea-f842-4bdd-bd11-ecb965eb5a78\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9855767cc8352d9cd036b2b8755bf1d71179ea38a63cb4113a60669da8d8556\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2815, \"end_char_idx\": 5679, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "ref_doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "chroma:document": "## Usage example\n\n### Single Media Mode\n\nThe model can accept both images and videos as input. Here's an example code for inference in half-precision (`torch.float16`):\n\n```python\nimport av\nimport torch\nimport numpy as np\nfrom transformers import VideoLlavaForConditionalGeneration, VideoLlavaProcessor\n\ndef read_video_pyav(container, indices):\n    '''\n    Decode the video with PyAV decoder.\n    Args:\n        container (`av.container.input.InputContainer`): PyAV container.\n        indices (`List[int]`): List of frame indices to decode.\n    Returns:\n        result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).\n    '''\n    frames = []\n    container.seek(0)\n    start_index = indices[0]\n    end_index = indices[-1]\n    for i, frame in enumerate(container.decode(video=0)):\n        if i > end_index:\n            break\n        if i >= start_index and i in indices:\n            frames.append(frame)\n    return np.stack([x.to_ndarray(format=\"rgb24\") for x in frames])\n\n# Load the model in half-precision\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\", torch_dtype=torch.float16, device_map=\"auto\")\nprocessor = VideoLlavaProcessor.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\")\n\n# Load the video as an np.arrau, sampling uniformly 8 frames\nvideo_path = hf_hub_download(repo_id=\"raushan-testing-hf/videos-test\", filename=\"sample_demo_1.mp4\", repo_type=\"dataset\")\ncontainer = av.open(video_path)\ntotal_frames = container.streams.video[0].frames\nindices = np.arange(0, total_frames, total_frames / 8).astype(int)\nvideo = read_video_pyav(container, indices)\n\n# For better results, we recommend to prompt the model in the following format\nprompt = \"USER: <video>\\nWhy is this funny? ASSISTANT:\"\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```\n\nFor multiple turns conversation change the prompt format to:\n\n```bash\n\"USER: <video>\\nWhat do you see in this video? ASSISTANT: A baby reading a book. USER: Why is the it funny? ASSISTANT:\"\n```\n\n### Mixed Media Mode\n\nThe model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: \n\n```python\nfrom PIL import Image\nimport requests\n\n# Generate from image and video mixed inputs\n# Load and image and write a new prompt\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"USER: <image>\\nHow many cats are there in the image? ASSISTANT: There are two cats. USER: <video>\\nWhy is this video funny?"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 3d4355ea-f842-4bdd-bd11-ecb965eb5a78
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"3d4355ea-f842-4bdd-bd11-ecb965eb5a78\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b8cc3e86-bca2-43dd-9c29-3b8e14f4e083\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a0b992f56c7b17508f2a9e9e451bff8068f4fc27d81ff60ec3863b9fa4e40a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c73fb89-6d56-4f15-a8c6-d8f2588c5eb2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8d277fc84412eacd3acc57762c51c657b5f2fcb858012a459157d31773f5262f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"15fba6cc-ae47-4f3f-8bd4-6489897e302f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0958f432101801dca225bcf8728ea85bc59940e6ac73b194fa35ee49e6ced23\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4574, \"end_char_idx\": 6941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "ref_doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "chroma:document": "ASSISTANT:\"\ninputs = processor(text=prompt, videos=video, return_tensors=\"pt\")\n\nout = model.generate(**inputs, max_new_tokens=60)\nprocessor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n```\n\nFor multiple turns conversation change the prompt format to:\n\n```bash\n\"USER: <video>\\nWhat do you see in this video? ASSISTANT: A baby reading a book. USER: Why is the it funny? ASSISTANT:\"\n```\n\n### Mixed Media Mode\n\nThe model can also generate from an interleaved image-video inputs. However note, that it was not trained in interleaved image-video setting which might affect the performance. Below is an example usage for mixed media input, add the following lines to the above code snippet: \n\n```python\nfrom PIL import Image\nimport requests\n\n# Generate from image and video mixed inputs\n# Load and image and write a new prompt\nurl = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nprompt = \"USER: <image>\\nHow many cats are there in the image? ASSISTANT: There are two cats. USER: <video>\\nWhy is this video funny? ASSISTANT:\"\n\ninputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors=\"pt\")\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_length=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes for memory efficiency\n\nThe model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. his allows for efficient deployment on resource-constrained cases. \n\nFirst make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:\n\n\n```python\nfrom transformers import VideoLlavaForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 15fba6cc-ae47-4f3f-8bd4-6489897e302f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/video_llava", "title": "Video-LLaVA", "tokens": 2128, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"15fba6cc-ae47-4f3f-8bd4-6489897e302f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b8cc3e86-bca2-43dd-9c29-3b8e14f4e083\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"f4a0b992f56c7b17508f2a9e9e451bff8068f4fc27d81ff60ec3863b9fa4e40a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3d4355ea-f842-4bdd-bd11-ecb965eb5a78\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/video_llava\", \"title\": \"Video-LLaVA\", \"tokens\": 2128, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"0b8a376337e63a2b43b65ea1ebc47078c68d7fec7cfe301d8067db2ef6b10071\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5609, \"end_char_idx\": 8309, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "ref_doc_id": "b8cc3e86-bca2-43dd-9c29-3b8e14f4e083", "chroma:document": "ASSISTANT: There are two cats. USER: <video>\\nWhy is this video funny? ASSISTANT:\"\n\ninputs = processor(text=prompt, images=image, videos=clip, padding=True, return_tensors=\"pt\")\n\n# Generate\ngenerate_ids = model.generate(**inputs, max_length=50)\nprocessor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n\n```\n\n## Model optimization\n\n### Quantization using Bitsandbytes for memory efficiency\n\nThe model can be loaded in lower bits, significantly reducing memory burden while maintaining the performance of the original model. his allows for efficient deployment on resource-constrained cases. \n\nFirst make sure to install bitsandbytes by running `pip install bitsandbytes` and to have access to a CUDA compatible GPU device. Load the quantized model by simply adding [`BitsAndBytesConfig`](../main_classes/quantization#transformers.BitsAndBytesConfig) as shown below:\n\n\n```python\nfrom transformers import VideoLlavaForConditionalGeneration, BitsAndBytesConfig\n\n# specify how to quantize the model\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\"LanguageBind/Video-LLaVA-7B-hf\", quantization_config=quantization_config, device_map=\"auto\")\n```\n\n\n### Flash-Attention 2 to speed-up generation\n\nAdditionally, we can greatly speed-up model inference by using [Flash Attention](../perf_train_gpu_one.md#flash-attention-2), which is a faster implementation of the attention mechanism used inside the model.\n\nFirst, make sure to install the latest version of Flash Attention 2:\n\n```bash\npip install -U flash-attn --no-build-isolation\n```\n\nAlso, you should have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`.\n\nTo load and run a model using Flash Attention-2, simply add `attn_implementation=\"flash_attention_2\"` when loading the model as follows:\n\n```python\nfrom transformers import VideoLlavaForConditionalGeneration\n\nmodel = VideoLlavaForConditionalGeneration.from_pretrained(\n    \"LanguageBind/Video-LLaVA-7B-hf\", \n    torch_dtype=torch.float16, \n    attn_implementation=\"flash_attention_2\",\n).to(0)\n```\n\n\n## VideoLlavaConfig\n\n[[autodoc]] VideoLlavaConfig\n\n## VideoLlavaImageProcessor\n\n[[autodoc]] VideoLlavaImageProcessor\n\n## VideoLlavaProcessor\n\n[[autodoc]] VideoLlavaProcessor\n\n## VideoLlavaForConditionalGeneration\n\n[[autodoc]] VideoLlavaForConditionalGeneration\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 2f290365-d5a1-4200-a74c-87f1013e072a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"2f290365-d5a1-4200-a74c-87f1013e072a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5f93dac0-c944-49d2-bde5-1fdc084bcc10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ea607fd76a21bfee6f45a446b70391f81fc20a4ce86619e73e4ac2d157080f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"58222dbd-3ed4-4505-b35d-d97328b41651\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"effd02f5291e1b8394a3f26c9c38728c1bce69dcc2f46c046297a53b881f3565\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3236, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5f93dac0-c944-49d2-bde5-1fdc084bcc10", "doc_id": "5f93dac0-c944-49d2-bde5-1fdc084bcc10", "ref_doc_id": "5f93dac0-c944-49d2-bde5-1fdc084bcc10", "chroma:document": "# MGP-STR\n\n## Overview\n\nThe MGP-STR model was proposed in [Multi-Granularity Prediction for Scene Text Recognition](https://arxiv.org/abs/2209.03592) by Peng Wang, Cheng Da, and Cong Yao. MGP-STR is a conceptually **simple** yet **powerful** vision Scene Text Recognition (STR) model, which is built upon the [Vision Transformer (ViT)](vit). To integrate linguistic knowledge, Multi-Granularity Prediction (MGP) strategy is proposed to inject information from the language modality into the model in an implicit way.\n\nThe abstract from the paper is the following:\n\n*Scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this challenging problem, numerous innovative methods have been successively proposed and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet powerful vision STR model, which is built upon ViT and outperforms previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely-used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. The resultant algorithm (termed MGP-STR) is able to push the performance envelop of STR to an even higher level. Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mgp_str_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> MGP-STR architecture. Taken from the <a href=\"https://arxiv.org/abs/2209.03592\">original paper</a>. </small>\n\nMGP-STR is trained on two synthetic datasets [MJSynth]((http://www.robots.ox.ac.uk/~vgg/data/text/)) (MJ) and [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE).\nThis model was contributed by [yuekun](https://huggingface.co/yuekun). The original code can be found [here](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).\n\n## Inference example\n\n[`MgpstrModel`] accepts images as input and generates three types of predictions, which represent textual information at different granularities.\nThe three types of predictions are fused to give the final prediction result.\n\nThe [`ViTImageProcessor`] class is responsible for preprocessing the input image and\n[`MgpstrTokenizer`] decodes the generated character tokens to the target string. The\n[`MgpstrProcessor`] wraps [`ViTImageProcessor`] and [`MgpstrTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 58222dbd-3ed4-4505-b35d-d97328b41651
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mgp-str", "title": "MGP-STR", "tokens": 1183, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"58222dbd-3ed4-4505-b35d-d97328b41651\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5f93dac0-c944-49d2-bde5-1fdc084bcc10\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"5ea607fd76a21bfee6f45a446b70391f81fc20a4ce86619e73e4ac2d157080f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2f290365-d5a1-4200-a74c-87f1013e072a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mgp-str\", \"title\": \"MGP-STR\", \"tokens\": 1183, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"8e0e373293980d46266a8c435d89c593de00329a2c0cd2781ce567a74d162146\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1691, \"end_char_idx\": 4290, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5f93dac0-c944-49d2-bde5-1fdc084bcc10", "doc_id": "5f93dac0-c944-49d2-bde5-1fdc084bcc10", "ref_doc_id": "5f93dac0-c944-49d2-bde5-1fdc084bcc10", "chroma:document": "Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks.*\n\n<img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/mgp_str_architecture.png\"\nalt=\"drawing\" width=\"600\"/>\n\n<small> MGP-STR architecture. Taken from the <a href=\"https://arxiv.org/abs/2209.03592\">original paper</a>. </small>\n\nMGP-STR is trained on two synthetic datasets [MJSynth]((http://www.robots.ox.ac.uk/~vgg/data/text/)) (MJ) and [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST) without fine-tuning on other datasets. It achieves state-of-the-art results on six standard Latin scene text benchmarks, including 3 regular text datasets (IC13, SVT, IIIT) and 3 irregular ones (IC15, SVTP, CUTE).\nThis model was contributed by [yuekun](https://huggingface.co/yuekun). The original code can be found [here](https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR).\n\n## Inference example\n\n[`MgpstrModel`] accepts images as input and generates three types of predictions, which represent textual information at different granularities.\nThe three types of predictions are fused to give the final prediction result.\n\nThe [`ViTImageProcessor`] class is responsible for preprocessing the input image and\n[`MgpstrTokenizer`] decodes the generated character tokens to the target string. The\n[`MgpstrProcessor`] wraps [`ViTImageProcessor`] and [`MgpstrTokenizer`]\ninto a single instance to both extract the input features and decode the predicted token ids.\n\n- Step-by-step Optical Character Recognition (OCR)\n\n```py\n>>> from transformers import MgpstrProcessor, MgpstrForSceneTextRecognition\n>>> import requests\n>>> from PIL import Image\n\n>>> processor = MgpstrProcessor.from_pretrained('alibaba-damo/mgp-str-base')\n>>> model = MgpstrForSceneTextRecognition.from_pretrained('alibaba-damo/mgp-str-base')\n\n>>> # load image from the IIIT-5k dataset\n>>> url = \"https://i.postimg.cc/ZKwLg2Gw/367-14.png\"\n>>> image = Image.open(requests.get(url, stream=True).raw).convert(\"RGB\")\n\n>>> pixel_values = processor(images=image, return_tensors=\"pt\").pixel_values\n>>> outputs = model(pixel_values)\n\n>>> generated_text = processor.batch_decode(outputs.logits)['generated_text']\n```\n\n## MgpstrConfig\n\n[[autodoc]] MgpstrConfig\n\n## MgpstrTokenizer\n\n[[autodoc]] MgpstrTokenizer\n    - save_vocabulary\n\n## MgpstrProcessor\n\n[[autodoc]] MgpstrProcessor\n    - __call__\n    - batch_decode\n\n## MgpstrModel\n\n[[autodoc]] MgpstrModel\n    - forward\n\n## MgpstrForSceneTextRecognition\n\n[[autodoc]] MgpstrForSceneTextRecognition\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0707330d-53cd-483c-a0dd-8f0d34b0629c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mega", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0707330d-53cd-483c-a0dd-8f0d34b0629c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7b278f5-4f87-467f-a388-9b830fd4faf9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7051f67626829bc5d4c487a754cae9a96855f7f99c23bc046f51909c683f7bb1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"619e616a-19d6-40d1-9067-d934d4faeb49\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5efce74adbd19bf6f2effa81a4a3e3741e2f9e898c475ccbda2f5df62728a6f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3237, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7b278f5-4f87-467f-a388-9b830fd4faf9", "doc_id": "a7b278f5-4f87-467f-a388-9b830fd4faf9", "ref_doc_id": "a7b278f5-4f87-467f-a388-9b830fd4faf9", "chroma:document": "# MEGA\n\n<Tip warning={true}>\n\nThis model is in maintenance mode only, we don't accept any new PRs changing its code.\nIf you run into any issues running this model, please reinstall the last version that supported this model: v4.40.2.\nYou can do so by running the following command: `pip install -U transformers==4.40.2`.\n\n</Tip>\n\n## Overview\n\nThe MEGA model was proposed in [Mega: Moving Average Equipped Gated Attention](https://arxiv.org/abs/2209.10655) by Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.\nMEGA proposes a new approach to self-attention with each encoder layer having a multi-headed exponential moving average in addition to a single head of standard dot-product attention, giving the attention mechanism\nstronger positional biases. This allows MEGA to perform competitively to Transformers on standard benchmarks including LRA\nwhile also having significantly fewer parameters. MEGA's compute efficiency allows it to scale to very long sequences, making it an\nattractive option for long-document NLP tasks.\n\nThe abstract from the paper is the following:\n\n *The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of Mega that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that Mega achieves significant improvements over other sequence models, including variants of Transformers and recent state space models. *\n\nThis model was contributed by [mnaylor](https://huggingface.co/mnaylor).\nThe original code can be found [here](https://github.com/facebookresearch/mega).\n\n\n## Usage tips\n\n- MEGA can perform quite well with relatively few parameters. See Appendix D in the MEGA paper for examples of architectural specs which perform well in various settings. If using MEGA as a decoder, be sure to set `bidirectional=False` to avoid errors with default bidirectional.\n- Mega-chunk is a variant of mega that reduces time and spaces complexity from quadratic to linear. Utilize chunking with MegaConfig.use_chunking and control chunk size with MegaConfig.chunk_size\n\n\n## Implementation Notes\n\n- The original implementation of MEGA had an inconsistent expectation of attention masks for padding and causal self-attention between the softmax attention and Laplace/squared ReLU method. This implementation addresses that inconsistency.\n- The original implementation did not include token type embeddings; this implementation adds support for these, with the option controlled by MegaConfig.add_token_type_embeddings"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 619e616a-19d6-40d1-9067-d934d4faeb49
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/mega", "title": "MEGA", "tokens": 955, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"619e616a-19d6-40d1-9067-d934d4faeb49\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7b278f5-4f87-467f-a388-9b830fd4faf9\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"7051f67626829bc5d4c487a754cae9a96855f7f99c23bc046f51909c683f7bb1\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0707330d-53cd-483c-a0dd-8f0d34b0629c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/mega\", \"title\": \"MEGA\", \"tokens\": 955, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"61242fe73f8e7597dc70ac7df1633554714127c22ffb93f22aca8b6f23886cd0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2300, \"end_char_idx\": 3786, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7b278f5-4f87-467f-a388-9b830fd4faf9", "doc_id": "a7b278f5-4f87-467f-a388-9b830fd4faf9", "ref_doc_id": "a7b278f5-4f87-467f-a388-9b830fd4faf9", "chroma:document": "## Usage tips\n\n- MEGA can perform quite well with relatively few parameters. See Appendix D in the MEGA paper for examples of architectural specs which perform well in various settings. If using MEGA as a decoder, be sure to set `bidirectional=False` to avoid errors with default bidirectional.\n- Mega-chunk is a variant of mega that reduces time and spaces complexity from quadratic to linear. Utilize chunking with MegaConfig.use_chunking and control chunk size with MegaConfig.chunk_size\n\n\n## Implementation Notes\n\n- The original implementation of MEGA had an inconsistent expectation of attention masks for padding and causal self-attention between the softmax attention and Laplace/squared ReLU method. This implementation addresses that inconsistency.\n- The original implementation did not include token type embeddings; this implementation adds support for these, with the option controlled by MegaConfig.add_token_type_embeddings\n\n\n## MegaConfig\n\n[[autodoc]] MegaConfig\n\n## MegaModel\n\n[[autodoc]] MegaModel\n    - forward\n\n## MegaForCausalLM\n\n[[autodoc]] MegaForCausalLM\n    - forward\n\n## MegaForMaskedLM\n\n[[autodoc]] MegaForMaskedLM\n    - forward\n\n## MegaForSequenceClassification\n\n[[autodoc]] MegaForSequenceClassification\n    - forward\n\n## MegaForMultipleChoice\n\n[[autodoc]] MegaForMultipleChoice\n    - forward\n\n## MegaForTokenClassification\n\n[[autodoc]] MegaForTokenClassification\n    - forward\n\n## MegaForQuestionAnswering\n\n[[autodoc]] MegaForQuestionAnswering\n    - forward"}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: 0eee6089-92a1-43b8-a88f-a8c22f1bc53c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/codegen", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"0eee6089-92a1-43b8-a88f-a8c22f1bc53c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"550e4a60-2442-4e59-a635-9e1fb69a6b26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20cb1150d528b9784b1960d0117988e39defea89acd257d14357396751b2f30e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d90734ea-4870-4514-a1a1-0bf64ab7a290\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6e6b44f9559abe8059e29715d25697fef0aa9ce310414362715890a42d84629\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2841, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "550e4a60-2442-4e59-a635-9e1fb69a6b26", "doc_id": "550e4a60-2442-4e59-a635-9e1fb69a6b26", "ref_doc_id": "550e4a60-2442-4e59-a635-9e1fb69a6b26", "chroma:document": "# CodeGen\n\n## Overview\n\nThe CodeGen model was proposed in [A Conversational Paradigm for Program Synthesis](https://arxiv.org/abs/2203.13474) by Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.\n\nCodeGen is an autoregressive language model for program synthesis trained sequentially on [The Pile](https://pile.eleuther.ai/), BigQuery, and BigPython.\n\nThe abstract from the paper is the following:\n\n*Program synthesis strives to generate a computer program as a solution to a given problem specification. We propose a conversational program synthesis approach via large language models, which addresses the challenges of searching over a vast program space and user intent specification faced in prior approaches. Our new approach casts the process of writing a specification and program as a multi-turn conversation between a user and a system. It treats program synthesis as a sequence prediction problem, in which the specification is expressed in natural language and the desired program is conditionally sampled. We train a family of large language models, called CodeGen, on natural language and programming language data. With weak supervision in the data and the scaling up of data size and model size, conversational capacities emerge from the simple autoregressive language modeling. To study the model behavior on conversational program synthesis, we develop a multi-turn programming benchmark (MTPB), where solving each problem requires multi-step synthesis via multi-turn conversation between the user and the model. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm. In addition, our model CodeGen (with up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the HumanEval benchmark. We make the training library JaxFormer including checkpoints available as open source contribution: [this https URL](https://github.com/salesforce/codegen).* \n\nThis model was contributed by [Hiroaki Hayashi](https://huggingface.co/rooa).\nThe original code can be found [here](https://github.com/salesforce/codegen).\n\n## Checkpoint Naming\n\n* CodeGen model [checkpoints](https://huggingface.co/models?other=codegen) are available on different pre-training data with variable sizes.\n* The format is: `Salesforce/codegen-{size}-{data}`, where\n  * `size`: `350M`, `2B`, `6B`, `16B`\n  * `data`: \n    * `nl`: Pre-trained on the Pile\n    * `multi`: Initialized with `nl`, then further pre-trained on multiple programming languages data\n    * `mono`: Initialized with `multi`, then further pre-trained on Python data\n* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python."}


created_at: 2024-08-26 20:30:03
topic: persistent://default/default/621deaf2-9359-4ac4-ada0-047176703e13
id: d90734ea-4870-4514-a1a1-0bf64ab7a290
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/transformers/model_doc/codegen", "title": "CodeGen", "tokens": 1009, "retrieve_doc": true, "source": "transformers", "_node_content": "{\"id_\": \"d90734ea-4870-4514-a1a1-0bf64ab7a290\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"550e4a60-2442-4e59-a635-9e1fb69a6b26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"20cb1150d528b9784b1960d0117988e39defea89acd257d14357396751b2f30e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0eee6089-92a1-43b8-a88f-a8c22f1bc53c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/transformers/model_doc/codegen\", \"title\": \"CodeGen\", \"tokens\": 1009, \"retrieve_doc\": true, \"source\": \"transformers\"}, \"hash\": \"02f63430ea6d6e3ef0ad94fff502a799e624ca80bbfa9c2b8fb9a34b0bd9e63b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1076, \"end_char_idx\": 3755, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "550e4a60-2442-4e59-a635-9e1fb69a6b26", "doc_id": "550e4a60-2442-4e59-a635-9e1fb69a6b26", "ref_doc_id": "550e4a60-2442-4e59-a635-9e1fb69a6b26", "chroma:document": "We train a family of large language models, called CodeGen, on natural language and programming language data. With weak supervision in the data and the scaling up of data size and model size, conversational capacities emerge from the simple autoregressive language modeling. To study the model behavior on conversational program synthesis, we develop a multi-turn programming benchmark (MTPB), where solving each problem requires multi-step synthesis via multi-turn conversation between the user and the model. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm. In addition, our model CodeGen (with up to 16B parameters trained on TPU-v4) outperforms OpenAI's Codex on the HumanEval benchmark. We make the training library JaxFormer including checkpoints available as open source contribution: [this https URL](https://github.com/salesforce/codegen).* \n\nThis model was contributed by [Hiroaki Hayashi](https://huggingface.co/rooa).\nThe original code can be found [here](https://github.com/salesforce/codegen).\n\n## Checkpoint Naming\n\n* CodeGen model [checkpoints](https://huggingface.co/models?other=codegen) are available on different pre-training data with variable sizes.\n* The format is: `Salesforce/codegen-{size}-{data}`, where\n  * `size`: `350M`, `2B`, `6B`, `16B`\n  * `data`: \n    * `nl`: Pre-trained on the Pile\n    * `multi`: Initialized with `nl`, then further pre-trained on multiple programming languages data\n    * `mono`: Initialized with `multi`, then further pre-trained on Python data\n* For example, `Salesforce/codegen-350M-mono` offers a 350 million-parameter checkpoint pre-trained sequentially on the Pile, multiple programming languages, and Python.\n\n## Usage example\n\n```python\n>>> from transformers import AutoModelForCausalLM, AutoTokenizer\n\n>>> checkpoint = \"Salesforce/codegen-350M-mono\"\n>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)\n>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n\n>>> text = \"def hello_world():\"\n\n>>> completion = model.generate(**tokenizer(text, return_tensors=\"pt\"))\n\n>>> print(tokenizer.decode(completion[0]))\ndef hello_world():\n    print(\"Hello World\")\n\nhello_world()\n```\n\n## Resources\n\n- [Causal language modeling task guide](../tasks/language_modeling)\n\n## CodeGenConfig\n\n[[autodoc]] CodeGenConfig\n    - all\n\n## CodeGenTokenizer\n\n[[autodoc]] CodeGenTokenizer\n    - create_token_type_ids_from_sequences\n    - save_vocabulary\n\n## CodeGenTokenizerFast\n\n[[autodoc]] CodeGenTokenizerFast\n\n## CodeGenModel\n\n[[autodoc]] CodeGenModel\n    - forward\n\n## CodeGenForCausalLM\n\n[[autodoc]] CodeGenForCausalLM\n    - forward"}


